ConcurrentRebalancersTest.TwoConcurrentRebalancers/1: /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/mini-cluster/external_mini_cluster.cc:947: Failure
Expected equality of these values:
  0
  num_crashes
    Which is: 2
At least one process crashed
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tools/rebalancer_tool-test.cc:1284: Failure
Expected: cluster_->AssertNoCrashes() doesn't generate new fatal failures in the current thread.
  Actual: it does.
I20240502 01:54:49.695906  9923 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu pid 17416
I20240502 01:54:49.703599 17538 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 01:54:49.762305 17832 consensus_peers.cc:485] T 17365abbb6c1430aa278719eb8e4a648 P da3584f429104d23a1a8899b35cc2138 -> Peer 29806d3d37354c169484f6db368f8798 (127.9.176.197:35011): Couldn't send request to peer 29806d3d37354c169484f6db368f8798. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.9.176.197:35011: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:54:49.797940 17565 consensus_peers.cc:462] T b40f968655774ce887dff8b7f4651501 P 1b50a031f2784ba498b3e52e16916205 -> Peer 29806d3d37354c169484f6db368f8798 (127.9.176.197:35011): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.9.176.197:35011: connect: Connection refused (error 111)
I20240502 01:54:49.983006  9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu with pid 17416
W20240502 01:54:50.018100 17832 consensus_peers.cc:485] T 3b3fa060e8d64911bec810ec07a358d2 P da3584f429104d23a1a8899b35cc2138 -> Peer 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198:36865): Couldn't send request to peer 4b7c382c9edd41babdd1a1afc904a813. Status: Network error: Client connection negotiation failed: client connection to 127.9.176.198:36865: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:54:50.029814 17568 connection.cc:535] client connection to 127.9.176.193:35397 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:54:50.030126 17703 connection.cc:535] client connection to 127.9.176.193:35397 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 01:54:50.030597  9923 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu pid 17549
W20240502 01:54:50.031467 17703 consensus_peers.cc:485] T d022e878d5d943eb849e53d15029aa77 P 84d3a4b174c34e02aedab1e620731359 -> Peer 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193:35397): Couldn't send request to peer 311f765f59754bdcb4eadc04e14bd4ea. Status: Network error: Client connection negotiation failed: client connection to 127.9.176.193:35397: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:50.031466 17568 consensus_peers.cc:485] T 38a60d7db47149ff90444e5276fc8935 P 1b50a031f2784ba498b3e52e16916205 -> Peer 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193:35397): Couldn't send request to peer 311f765f59754bdcb4eadc04e14bd4ea. Status: Network error: Client connection negotiation failed: client connection to 127.9.176.193:35397: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:50.043258 17674 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 01:54:50.048134 17703 consensus_peers.cc:485] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 84d3a4b174c34e02aedab1e620731359 -> Peer 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193:35397): Couldn't send request to peer 311f765f59754bdcb4eadc04e14bd4ea. Status: Network error: Client connection negotiation failed: client connection to 127.9.176.193:35397: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:50.097956 17832 consensus_peers.cc:485] T 05b67d9912f047bfb6a2bed90b82ae15 P da3584f429104d23a1a8899b35cc2138 -> Peer 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198:36865): Couldn't send request to peer 4b7c382c9edd41babdd1a1afc904a813. Status: Network error: Client connection negotiation failed: client connection to 127.9.176.198:36865: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:54:50.193208 17700 consensus_peers.cc:485] T c91f74f46b04425aa5393515cb9f781c P 84d3a4b174c34e02aedab1e620731359 -> Peer 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198:36865): Couldn't send request to peer 4b7c382c9edd41babdd1a1afc904a813. Status: Network error: Client connection negotiation failed: client connection to 127.9.176.198:36865: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 01:54:50.331667  9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu with pid 17549
W20240502 01:54:50.385378 19806 connection.cc:535] server connection from 127.9.176.194:49149 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 01:54:50.386035  9923 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu pid 17684
I20240502 01:54:50.393234 17806 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 01:54:50.476048 17835 consensus_peers.cc:485] T 3bdd49939e994240a6d15dbd11559ff4 P da3584f429104d23a1a8899b35cc2138 -> Peer 1b50a031f2784ba498b3e52e16916205 (127.9.176.194:45617): Couldn't send request to peer 1b50a031f2784ba498b3e52e16916205. Status: Network error: Client connection negotiation failed: client connection to 127.9.176.194:45617: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:50.670783  9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu with pid 17684
W20240502 01:54:50.715981 19669 connection.cc:535] server connection from 127.9.176.195:60725 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:54:50.716632 17342 connection.cc:535] server connection from 127.9.176.195:60083 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 01:54:50.716955  9923 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu pid 17817
I20240502 01:54:50.723198 17941 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:54:50.798244 22948 raft_consensus.cc:482] T 075ded9523e44c3d92844805338643a3 P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 311f765f59754bdcb4eadc04e14bd4ea)
I20240502 01:54:50.798570 22948 raft_consensus.cc:504] T 075ded9523e44c3d92844805338643a3 P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } }
I20240502 01:54:50.800264 22948 leader_election.cc:288] T 075ded9523e44c3d92844805338643a3 P 94931f4b51c9493cb3d4264ce99f5e76 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1b50a031f2784ba498b3e52e16916205 (127.9.176.194:45617), 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193:35397)
W20240502 01:54:50.801738 19806 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.176.194:45617: connect: Connection refused (error 111)
W20240502 01:54:50.803315 19806 leader_election.cc:334] T 075ded9523e44c3d92844805338643a3 P 94931f4b51c9493cb3d4264ce99f5e76 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1b50a031f2784ba498b3e52e16916205 (127.9.176.194:45617): Network error: Client connection negotiation failed: client connection to 127.9.176.194:45617: connect: Connection refused (error 111)
W20240502 01:54:50.803586 19806 leader_election.cc:334] T 075ded9523e44c3d92844805338643a3 P 94931f4b51c9493cb3d4264ce99f5e76 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193:35397): Network error: Client connection negotiation failed: client connection to 127.9.176.193:35397: connect: Connection refused (error 111)
I20240502 01:54:50.803726 19806 leader_election.cc:302] T 075ded9523e44c3d92844805338643a3 P 94931f4b51c9493cb3d4264ce99f5e76 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94931f4b51c9493cb3d4264ce99f5e76; no voters: 1b50a031f2784ba498b3e52e16916205, 311f765f59754bdcb4eadc04e14bd4ea
I20240502 01:54:50.804083 22948 raft_consensus.cc:2726] T 075ded9523e44c3d92844805338643a3 P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:54:50.982318 22953 raft_consensus.cc:482] T fadcc8fe6ae047c1b142514aad8bebba P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 311f765f59754bdcb4eadc04e14bd4ea)
I20240502 01:54:50.982636 22953 raft_consensus.cc:504] T fadcc8fe6ae047c1b142514aad8bebba P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } }
I20240502 01:54:50.983896 22953 leader_election.cc:288] T fadcc8fe6ae047c1b142514aad8bebba P da3584f429104d23a1a8899b35cc2138 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193:35397), 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198:36865)
I20240502 01:54:50.984131 22953 raft_consensus.cc:482] T 37eda67598fe491bbb29aa62b2c4bec7 P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 311f765f59754bdcb4eadc04e14bd4ea)
I20240502 01:54:50.984342 22953 raft_consensus.cc:504] T 37eda67598fe491bbb29aa62b2c4bec7 P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { promote: false } }
I20240502 01:54:50.985208 22953 leader_election.cc:288] T 37eda67598fe491bbb29aa62b2c4bec7 P da3584f429104d23a1a8899b35cc2138 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193:35397), 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200:37831)
I20240502 01:54:50.985893 19872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37eda67598fe491bbb29aa62b2c4bec7" candidate_uuid: "da3584f429104d23a1a8899b35cc2138" candidate_term: 2 candidate_status { last_received { term: 1 index: 14 } } ignore_live_leader: false dest_uuid: "94931f4b51c9493cb3d4264ce99f5e76" is_pre_election: true
I20240502 01:54:50.986822  9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu with pid 17817
I20240502 01:54:50.999555 22948 raft_consensus.cc:482] T 37eda67598fe491bbb29aa62b2c4bec7 P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 311f765f59754bdcb4eadc04e14bd4ea)
I20240502 01:54:50.999826 22948 raft_consensus.cc:504] T 37eda67598fe491bbb29aa62b2c4bec7 P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { promote: false } }
I20240502 01:54:51.000768 22948 leader_election.cc:288] T 37eda67598fe491bbb29aa62b2c4bec7 P 94931f4b51c9493cb3d4264ce99f5e76 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193:35397), da3584f429104d23a1a8899b35cc2138 (127.9.176.196:46381)
W20240502 01:54:51.002225 19806 leader_election.cc:334] T 37eda67598fe491bbb29aa62b2c4bec7 P 94931f4b51c9493cb3d4264ce99f5e76 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193:35397): Network error: Client connection negotiation failed: client connection to 127.9.176.193:35397: connect: Connection refused (error 111)
W20240502 01:54:51.025339 19803 connection.cc:535] client connection to 127.9.176.196:46381 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 01:54:51.026031  9923 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu pid 19654
W20240502 01:54:51.026542 19803 leader_election.cc:334] T 37eda67598fe491bbb29aa62b2c4bec7 P 94931f4b51c9493cb3d4264ce99f5e76 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer da3584f429104d23a1a8899b35cc2138 (127.9.176.196:46381): Network error: Client connection negotiation failed: client connection to 127.9.176.196:46381: connect: Connection refused (error 111)
I20240502 01:54:51.026712 19803 leader_election.cc:302] T 37eda67598fe491bbb29aa62b2c4bec7 P 94931f4b51c9493cb3d4264ce99f5e76 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94931f4b51c9493cb3d4264ce99f5e76; no voters: 311f765f59754bdcb4eadc04e14bd4ea, da3584f429104d23a1a8899b35cc2138
I20240502 01:54:51.027179 22948 raft_consensus.cc:2726] T 37eda67598fe491bbb29aa62b2c4bec7 P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:54:51.030521 22948 raft_consensus.cc:482] T 49ef898a43704b87ba077cd7b0485e9a P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 311f765f59754bdcb4eadc04e14bd4ea)
I20240502 01:54:51.030755 22948 raft_consensus.cc:504] T 49ef898a43704b87ba077cd7b0485e9a P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } }
I20240502 01:54:51.031836 22948 leader_election.cc:288] T 49ef898a43704b87ba077cd7b0485e9a P 94931f4b51c9493cb3d4264ce99f5e76 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193:35397), da3584f429104d23a1a8899b35cc2138 (127.9.176.196:46381), 29806d3d37354c169484f6db368f8798 (127.9.176.197:35011)
I20240502 01:54:51.033556 19778 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 01:54:51.033947 19803 leader_election.cc:334] T 49ef898a43704b87ba077cd7b0485e9a P 94931f4b51c9493cb3d4264ce99f5e76 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer da3584f429104d23a1a8899b35cc2138 (127.9.176.196:46381): Network error: Client connection negotiation failed: client connection to 127.9.176.196:46381: connect: Connection refused (error 111)
W20240502 01:54:51.034221 19806 leader_election.cc:334] T 49ef898a43704b87ba077cd7b0485e9a P 94931f4b51c9493cb3d4264ce99f5e76 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193:35397): Network error: Client connection negotiation failed: client connection to 127.9.176.193:35397: connect: Connection refused (error 111)
I20240502 01:54:51.034337 19806 leader_election.cc:302] T 49ef898a43704b87ba077cd7b0485e9a P 94931f4b51c9493cb3d4264ce99f5e76 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 94931f4b51c9493cb3d4264ce99f5e76; no voters: 311f765f59754bdcb4eadc04e14bd4ea, da3584f429104d23a1a8899b35cc2138
I20240502 01:54:51.034652 22948 raft_consensus.cc:2726] T 49ef898a43704b87ba077cd7b0485e9a P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 01:54:51.034711 19803 leader_election.cc:334] T 49ef898a43704b87ba077cd7b0485e9a P 94931f4b51c9493cb3d4264ce99f5e76 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29806d3d37354c169484f6db368f8798 (127.9.176.197:35011): Network error: Client connection negotiation failed: client connection to 127.9.176.197:35011: connect: Connection refused (error 111)
I20240502 01:54:51.125571 22948 raft_consensus.cc:482] T 62fd543d5c7c41f98926f2148ee61e7d P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 311f765f59754bdcb4eadc04e14bd4ea)
I20240502 01:54:51.125824 22948 raft_consensus.cc:504] T 62fd543d5c7c41f98926f2148ee61e7d P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } }
I20240502 01:54:51.126643 22948 leader_election.cc:288] T 62fd543d5c7c41f98926f2148ee61e7d P 94931f4b51c9493cb3d4264ce99f5e76 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193:35397), 272b2179c1b4476090a89880df2a8baa (127.9.176.199:42269)
W20240502 01:54:51.128096 19806 leader_election.cc:334] T 62fd543d5c7c41f98926f2148ee61e7d P 94931f4b51c9493cb3d4264ce99f5e76 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193:35397): Network error: Client connection negotiation failed: client connection to 127.9.176.193:35397: connect: Connection refused (error 111)
I20240502 01:54:51.230603 22963 raft_consensus.cc:482] T 62fd543d5c7c41f98926f2148ee61e7d P 272b2179c1b4476090a89880df2a8baa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 311f765f59754bdcb4eadc04e14bd4ea)
I20240502 01:54:51.230834  9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu with pid 19654
W20240502 01:54:51.265826 22949 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 01:54:51.126912 (+     0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.199:42269 (local address 127.9.176.200:56033)
0502 01:54:51.127148 (+   236us) negotiation.cc:107] Waiting for socket to connect
0502 01:54:51.127165 (+    17us) client_negotiation.cc:174] Beginning negotiation
0502 01:54:51.127289 (+   124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 01:54:51.264991 (+137702us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.199:42269: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":161}
I20240502 01:54:51.266512  9923 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu pid 19787
W20240502 01:54:51.267153 19803 leader_election.cc:334] T 62fd543d5c7c41f98926f2148ee61e7d P 94931f4b51c9493cb3d4264ce99f5e76 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 272b2179c1b4476090a89880df2a8baa (127.9.176.199:42269): Network error: Client connection negotiation failed: client connection to 127.9.176.199:42269: connect: Connection refused (error 111)
I20240502 01:54:51.267383 19803 leader_election.cc:302] T 62fd543d5c7c41f98926f2148ee61e7d P 94931f4b51c9493cb3d4264ce99f5e76 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94931f4b51c9493cb3d4264ce99f5e76; no voters: 272b2179c1b4476090a89880df2a8baa, 311f765f59754bdcb4eadc04e14bd4ea
I20240502 01:54:51.267861 22948 raft_consensus.cc:2726] T 62fd543d5c7c41f98926f2148ee61e7d P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:54:51.273329 19908 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:54:51.473692 22948 raft_consensus.cc:482] T 91e0e82526c0489a8908e250bf3c851f P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1b50a031f2784ba498b3e52e16916205)
I20240502 01:54:51.473997 22948 raft_consensus.cc:504] T 91e0e82526c0489a8908e250bf3c851f P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { promote: false } }
I20240502 01:54:51.474293 22967 raft_consensus.cc:482] T 2c78951695284b1aaab411c0a041c28a P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1b50a031f2784ba498b3e52e16916205)
I20240502 01:54:51.474539  9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu with pid 19787
I20240502 01:54:51.505049  9923 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu pid 17325
I20240502 01:54:51.506142 17385 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:54:51.767287  9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu with pid 17325
2024-05-02T01:54:51Z chronyd exiting
I20240502 01:54:51.831691  9923 test_util.cc:175] -----------------------------------------------
I20240502 01:54:51.831804  9923 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0

Note: This is test shard 4 of 8.
[==========] Running 7 tests from 6 test suites.
[----------] Global test environment set-up.
[----------] 1 test from LocationAwareRebalancingBasicTest
[ RUN      ] LocationAwareRebalancingBasicTest.Basic
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240502 01:52:31.051504  9923 test_util.cc:268] Using random seed: -618021183
I20240502 01:52:31.063548  9923 ts_itest-base.cc:112] Starting cluster with:
I20240502 01:52:31.063659  9923 ts_itest-base.cc:113] --------------
I20240502 01:52:31.063705  9923 ts_itest-base.cc:114] 6 tablet servers
I20240502 01:52:31.063768  9923 ts_itest-base.cc:115] 3 replicas per TS
I20240502 01:52:31.063833  9923 ts_itest-base.cc:116] --------------
2024-05-02T01:52:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T01:52:31Z Disabled control of system clock
I20240502 01:52:31.090862  9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
/tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:36089
--webserver_interface=127.9.176.254
--webserver_port=0
--builtin_ntp_servers=127.9.176.212:38509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.176.254:36089
--location_mapping_cmd=/tmp/dist-test-taskJZxTxO/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/location-assignment.state --map /A:2 --map /B:2 --map /C:2
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240502 01:52:31.422770  9936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:31.423122  9936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:52:31.423276  9936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:31.430706  9936 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:52:31.430861  9936 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 01:52:31.430954  9936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:52:31.431054  9936 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 01:52:31.431128  9936 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 01:52:31.440532  9936 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:38509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--location_mapping_cmd=/tmp/dist-test-taskJZxTxO/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/location-assignment.state --map /A:2 --map /B:2 --map /C:2
--ipki_ca_key_size=768
--master_addresses=127.9.176.254:36089
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:36089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.9.176.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true

Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:52:31.441979  9936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:52:31.443775  9936 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:31.454908  9941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:31.455475  9944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:31.455998  9942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:31.456724  9936 server_base.cc:1034] running on GCE node
I20240502 01:52:31.457612  9936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:52:31.459100  9936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:52:31.460320  9936 hybrid_clock.cc:648] HybridClock initialized: now 1714614751460270 us; error 42 us; skew 500 ppm
I20240502 01:52:31.460637  9936 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:31.467240  9936 webserver.cc:460] Webserver started at http://127.9.176.254:46365/ using document root <none> and password file <none>
I20240502 01:52:31.468139  9936 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:31.468252  9936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:31.468541  9936 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:52:31.471326  9936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "7684868bab274f87b883d33cfb6b8214"
format_stamp: "Formatted at 2024-05-02 01:52:31 on dist-test-slave-j3hh"
I20240502 01:52:31.472175  9936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "7684868bab274f87b883d33cfb6b8214"
format_stamp: "Formatted at 2024-05-02 01:52:31 on dist-test-slave-j3hh"
I20240502 01:52:31.477871  9936 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.000s
I20240502 01:52:31.481485  9950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:52:31.482923  9936 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20240502 01:52:31.483106  9936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/wal
uuid: "7684868bab274f87b883d33cfb6b8214"
format_stamp: "Formatted at 2024-05-02 01:52:31 on dist-test-slave-j3hh"
I20240502 01:52:31.483323  9936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:31.517989  9936 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:52:31.518927  9936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:52:31.519209  9936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:31.537961  9936 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.254:36089
I20240502 01:52:31.537981 10002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.254:36089 every 8 connection(s)
I20240502 01:52:31.539439  9936 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 01:52:31.543470 10003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:31.549990  9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu as pid 9936
I20240502 01:52:31.550311  9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 01:52:31.554980 10003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7684868bab274f87b883d33cfb6b8214: Bootstrap starting.
I20240502 01:52:31.559039 10003 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7684868bab274f87b883d33cfb6b8214: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:31.560339 10003 log.cc:824] T 00000000000000000000000000000000 P 7684868bab274f87b883d33cfb6b8214: Log is configured to *not* fsync() on all Append() calls
I20240502 01:52:31.562896 10003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7684868bab274f87b883d33cfb6b8214: No bootstrap required, opened a new log
I20240502 01:52:31.570602 10003 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7684868bab274f87b883d33cfb6b8214 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7684868bab274f87b883d33cfb6b8214" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 36089 } }
I20240502 01:52:31.571202 10003 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7684868bab274f87b883d33cfb6b8214 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:31.571367 10003 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7684868bab274f87b883d33cfb6b8214 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7684868bab274f87b883d33cfb6b8214, State: Initialized, Role: FOLLOWER
I20240502 01:52:31.572141 10003 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7684868bab274f87b883d33cfb6b8214 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7684868bab274f87b883d33cfb6b8214" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 36089 } }
I20240502 01:52:31.572376 10003 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7684868bab274f87b883d33cfb6b8214 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:52:31.572531 10003 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7684868bab274f87b883d33cfb6b8214 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:52:31.572779 10003 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7684868bab274f87b883d33cfb6b8214 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:31.576159 10003 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7684868bab274f87b883d33cfb6b8214 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7684868bab274f87b883d33cfb6b8214" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 36089 } }
I20240502 01:52:31.576875 10003 leader_election.cc:302] T 00000000000000000000000000000000 P 7684868bab274f87b883d33cfb6b8214 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7684868bab274f87b883d33cfb6b8214; no voters: 
I20240502 01:52:31.577663 10003 leader_election.cc:288] T 00000000000000000000000000000000 P 7684868bab274f87b883d33cfb6b8214 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240502 01:52:31.577818 10008 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7684868bab274f87b883d33cfb6b8214 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:31.580052 10003 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7684868bab274f87b883d33cfb6b8214 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 01:52:31.582060 10008 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7684868bab274f87b883d33cfb6b8214 [term 1 LEADER]: Becoming Leader. State: Replica: 7684868bab274f87b883d33cfb6b8214, State: Running, Role: LEADER
I20240502 01:52:31.582793 10008 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7684868bab274f87b883d33cfb6b8214 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7684868bab274f87b883d33cfb6b8214" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 36089 } }
I20240502 01:52:31.585764 10010 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7684868bab274f87b883d33cfb6b8214 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7684868bab274f87b883d33cfb6b8214. Latest consensus state: current_term: 1 leader_uuid: "7684868bab274f87b883d33cfb6b8214" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7684868bab274f87b883d33cfb6b8214" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 36089 } } }
I20240502 01:52:31.586089 10010 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7684868bab274f87b883d33cfb6b8214 [sys.catalog]: This master's current role is: LEADER
I20240502 01:52:31.586578 10009 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7684868bab274f87b883d33cfb6b8214 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7684868bab274f87b883d33cfb6b8214" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7684868bab274f87b883d33cfb6b8214" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 36089 } } }
I20240502 01:52:31.586810 10009 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7684868bab274f87b883d33cfb6b8214 [sys.catalog]: This master's current role is: LEADER
I20240502 01:52:31.589483 10018 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 01:52:31.597158 10018 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 01:52:31.607817 10018 catalog_manager.cc:1347] Generated new cluster ID: 79a462099ced4d6fafd13314f79e32b0
I20240502 01:52:31.607954 10018 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 01:52:31.621706 10018 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 01:52:31.623183 10018 catalog_manager.cc:1504] Loading token signing keys...
I20240502 01:52:31.635282 10018 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 7684868bab274f87b883d33cfb6b8214: Generated new TSK 0
I20240502 01:52:31.636605 10018 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 01:52:31.646056  9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
/tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.193:0
--local_ip_for_outbound_sockets=127.9.176.193
--webserver_interface=127.9.176.193
--webserver_port=0
--tserver_master_addrs=127.9.176.254:36089
--builtin_ntp_servers=127.9.176.212:38509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 01:52:31.983887 10028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:31.984252 10028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:52:31.984411 10028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:31.992295 10028 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:52:31.992452 10028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:52:31.992655 10028 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.193
I20240502 01:52:32.001950 10028 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:38509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.9.176.193
--webserver_port=0
--tserver_master_addrs=127.9.176.254:36089
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.193
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:52:32.003441 10028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:52:32.005311 10028 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:32.017314 10033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:32.017786 10034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:32.018302 10036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:32.018674 10028 server_base.cc:1034] running on GCE node
I20240502 01:52:32.019254 10028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:52:32.020174 10028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:52:32.021373 10028 hybrid_clock.cc:648] HybridClock initialized: now 1714614752021330 us; error 39 us; skew 500 ppm
I20240502 01:52:32.021679 10028 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:32.023993 10028 webserver.cc:460] Webserver started at http://127.9.176.193:35593/ using document root <none> and password file <none>
I20240502 01:52:32.024719 10028 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:32.024842 10028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:32.025161 10028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:52:32.027769 10028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "a885359a82bd4515913f7b273591f851"
format_stamp: "Formatted at 2024-05-02 01:52:32 on dist-test-slave-j3hh"
I20240502 01:52:32.030642 10028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "a885359a82bd4515913f7b273591f851"
format_stamp: "Formatted at 2024-05-02 01:52:32 on dist-test-slave-j3hh"
I20240502 01:52:32.036012 10028 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.000s
I20240502 01:52:32.039736 10042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:52:32.041198 10028 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20240502 01:52:32.041378 10028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "a885359a82bd4515913f7b273591f851"
format_stamp: "Formatted at 2024-05-02 01:52:32 on dist-test-slave-j3hh"
I20240502 01:52:32.041607 10028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:32.060735 10028 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:52:32.061642 10028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:52:32.061930 10028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:32.062875 10028 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:52:32.064478 10028 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:52:32.064582 10028 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:52:32.064709 10028 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:52:32.064785 10028 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:52:32.100210 10028 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.193:38373
I20240502 01:52:32.100262 10156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.193:38373 every 8 connection(s)
I20240502 01:52:32.101541 10028 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 01:52:32.105094  9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu as pid 10028
I20240502 01:52:32.105346  9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 01:52:32.108918  9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
/tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.194:0
--local_ip_for_outbound_sockets=127.9.176.194
--webserver_interface=127.9.176.194
--webserver_port=0
--tserver_master_addrs=127.9.176.254:36089
--builtin_ntp_servers=127.9.176.212:38509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:52:32.115267 10157 heartbeater.cc:347] Connected to a master server at 127.9.176.254:36089
I20240502 01:52:32.115607 10157 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:32.116484 10157 heartbeater.cc:511] Master 127.9.176.254:36089 requested a full tablet report, sending...
I20240502 01:52:32.176429  9967 ts_manager.cc:194] Registered new tserver with Master: a885359a82bd4515913f7b273591f851 (127.9.176.193:38373)
I20240502 01:52:32.178200  9967 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.176.193:57453
W20240502 01:52:32.455384 10161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:32.455781 10161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:52:32.455929 10161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:32.463794 10161 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:52:32.463958 10161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:52:32.464123 10161 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.194
I20240502 01:52:32.473546 10161 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:38509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.9.176.194
--webserver_port=0
--tserver_master_addrs=127.9.176.254:36089
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.194
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:52:32.474996 10161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:52:32.476922 10161 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:32.489291 10167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:32.489724 10168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:32.490231 10170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:32.490573 10161 server_base.cc:1034] running on GCE node
I20240502 01:52:32.491186 10161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:52:32.492228 10161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:52:32.493438 10161 hybrid_clock.cc:648] HybridClock initialized: now 1714614752493388 us; error 43 us; skew 500 ppm
I20240502 01:52:32.493791 10161 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:32.496186 10161 webserver.cc:460] Webserver started at http://127.9.176.194:42511/ using document root <none> and password file <none>
I20240502 01:52:32.496973 10161 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:32.497087 10161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:32.497424 10161 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:52:32.500140 10161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "7b7a47245f294ce9a264fcc8f6351c0f"
format_stamp: "Formatted at 2024-05-02 01:52:32 on dist-test-slave-j3hh"
I20240502 01:52:32.500928 10161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "7b7a47245f294ce9a264fcc8f6351c0f"
format_stamp: "Formatted at 2024-05-02 01:52:32 on dist-test-slave-j3hh"
I20240502 01:52:32.506381 10161 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.006s	sys 0.000s
I20240502 01:52:32.510016 10176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:52:32.511358 10161 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.001s
I20240502 01:52:32.511518 10161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "7b7a47245f294ce9a264fcc8f6351c0f"
format_stamp: "Formatted at 2024-05-02 01:52:32 on dist-test-slave-j3hh"
I20240502 01:52:32.511777 10161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:32.531877 10161 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:52:32.532716 10161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:52:32.533015 10161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:32.533959 10161 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:52:32.535537 10161 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:52:32.535660 10161 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.001s	sys 0.000s
I20240502 01:52:32.535749 10161 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:52:32.535805 10161 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:52:32.574762 10161 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.194:43645
I20240502 01:52:32.574812 10290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.194:43645 every 8 connection(s)
I20240502 01:52:32.576249 10161 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 01:52:32.578768  9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu as pid 10161
I20240502 01:52:32.578963  9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 01:52:32.582733  9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
/tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.195:0
--local_ip_for_outbound_sockets=127.9.176.195
--webserver_interface=127.9.176.195
--webserver_port=0
--tserver_master_addrs=127.9.176.254:36089
--builtin_ntp_servers=127.9.176.212:38509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:52:32.592223 10291 heartbeater.cc:347] Connected to a master server at 127.9.176.254:36089
I20240502 01:52:32.592592 10291 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:32.593480 10291 heartbeater.cc:511] Master 127.9.176.254:36089 requested a full tablet report, sending...
I20240502 01:52:32.646136  9967 ts_manager.cc:194] Registered new tserver with Master: 7b7a47245f294ce9a264fcc8f6351c0f (127.9.176.194:43645)
I20240502 01:52:32.647138  9967 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.176.194:33451
W20240502 01:52:32.929452 10294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:32.929769 10294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:52:32.929911 10294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:32.937239 10294 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:52:32.937386 10294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:52:32.937551 10294 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.195
I20240502 01:52:32.946825 10294 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:38509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.9.176.195
--webserver_port=0
--tserver_master_addrs=127.9.176.254:36089
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.195
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:52:32.948225 10294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:52:32.950135 10294 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:32.962806 10301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:32.963279 10300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:32.963658 10303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:32.964040 10294 server_base.cc:1034] running on GCE node
I20240502 01:52:32.964668 10294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:52:32.965658 10294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:52:32.966853 10294 hybrid_clock.cc:648] HybridClock initialized: now 1714614752966794 us; error 56 us; skew 500 ppm
I20240502 01:52:32.967157 10294 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:32.969465 10294 webserver.cc:460] Webserver started at http://127.9.176.195:33963/ using document root <none> and password file <none>
I20240502 01:52:32.970247 10294 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:32.970366 10294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:32.970731 10294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:52:32.973650 10294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "ede1dd598e3541138a19a3ea5c052bc9"
format_stamp: "Formatted at 2024-05-02 01:52:32 on dist-test-slave-j3hh"
I20240502 01:52:32.974514 10294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "ede1dd598e3541138a19a3ea5c052bc9"
format_stamp: "Formatted at 2024-05-02 01:52:32 on dist-test-slave-j3hh"
I20240502 01:52:32.980121 10294 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.004s	sys 0.003s
I20240502 01:52:32.983808 10309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:52:32.985354 10294 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20240502 01:52:32.985524 10294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "ede1dd598e3541138a19a3ea5c052bc9"
format_stamp: "Formatted at 2024-05-02 01:52:32 on dist-test-slave-j3hh"
I20240502 01:52:32.985745 10294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:33.008038 10294 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:52:33.008952 10294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:52:33.009232 10294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:33.010267 10294 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:52:33.011864 10294 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:52:33.011958 10294 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:52:33.012074 10294 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:52:33.012157 10294 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:52:33.046049 10294 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.195:38701
I20240502 01:52:33.046123 10423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.195:38701 every 8 connection(s)
I20240502 01:52:33.047466 10294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 01:52:33.053356  9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu as pid 10294
I20240502 01:52:33.053617  9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 01:52:33.057476  9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
/tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.196:0
--local_ip_for_outbound_sockets=127.9.176.196
--webserver_interface=127.9.176.196
--webserver_port=0
--tserver_master_addrs=127.9.176.254:36089
--builtin_ntp_servers=127.9.176.212:38509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:52:33.063191 10424 heartbeater.cc:347] Connected to a master server at 127.9.176.254:36089
I20240502 01:52:33.063526 10424 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:33.064473 10424 heartbeater.cc:511] Master 127.9.176.254:36089 requested a full tablet report, sending...
I20240502 01:52:33.108414  9967 ts_manager.cc:194] Registered new tserver with Master: ede1dd598e3541138a19a3ea5c052bc9 (127.9.176.195:38701)
I20240502 01:52:33.109347  9967 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.176.195:43201
I20240502 01:52:33.181538 10157 heartbeater.cc:503] Master 127.9.176.254:36089 was elected leader, sending a full tablet report...
W20240502 01:52:33.401399 10427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:33.401779 10427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:52:33.401926 10427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:33.409880 10427 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:52:33.410053 10427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:52:33.410250 10427 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.196
I20240502 01:52:33.419888 10427 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:38509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.9.176.196
--webserver_port=0
--tserver_master_addrs=127.9.176.254:36089
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.196
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:52:33.421291 10427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:52:33.423053 10427 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:33.435333 10433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:33.435932 10434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:33.436297 10436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:33.436658 10427 server_base.cc:1034] running on GCE node
I20240502 01:52:33.437297 10427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:52:33.438346 10427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:52:33.439548 10427 hybrid_clock.cc:648] HybridClock initialized: now 1714614753439482 us; error 59 us; skew 500 ppm
I20240502 01:52:33.439966 10427 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:33.442479 10427 webserver.cc:460] Webserver started at http://127.9.176.196:33535/ using document root <none> and password file <none>
I20240502 01:52:33.443280 10427 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:33.443408 10427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:33.443800 10427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:52:33.446693 10427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "34da849acb2045a9a2ef03933592dd08"
format_stamp: "Formatted at 2024-05-02 01:52:33 on dist-test-slave-j3hh"
I20240502 01:52:33.447549 10427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "34da849acb2045a9a2ef03933592dd08"
format_stamp: "Formatted at 2024-05-02 01:52:33 on dist-test-slave-j3hh"
I20240502 01:52:33.453547 10427 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.004s	sys 0.000s
I20240502 01:52:33.457656 10442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:52:33.459362 10427 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.000s
I20240502 01:52:33.459554 10427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "34da849acb2045a9a2ef03933592dd08"
format_stamp: "Formatted at 2024-05-02 01:52:33 on dist-test-slave-j3hh"
I20240502 01:52:33.459818 10427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:33.483705 10427 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:52:33.484721 10427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:52:33.485039 10427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:33.486202 10427 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:52:33.487996 10427 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:52:33.488098 10427 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:52:33.488216 10427 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:52:33.488281 10427 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:52:33.520893 10427 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.196:44521
I20240502 01:52:33.520941 10557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.196:44521 every 8 connection(s)
I20240502 01:52:33.522290 10427 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 01:52:33.527953  9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu as pid 10427
I20240502 01:52:33.528120  9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 01:52:33.531695  9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
/tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.197:0
--local_ip_for_outbound_sockets=127.9.176.197
--webserver_interface=127.9.176.197
--webserver_port=0
--tserver_master_addrs=127.9.176.254:36089
--builtin_ntp_servers=127.9.176.212:38509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:52:33.536877 10558 heartbeater.cc:347] Connected to a master server at 127.9.176.254:36089
I20240502 01:52:33.537235 10558 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:33.538427 10558 heartbeater.cc:511] Master 127.9.176.254:36089 requested a full tablet report, sending...
I20240502 01:52:33.591305  9967 ts_manager.cc:194] Registered new tserver with Master: 34da849acb2045a9a2ef03933592dd08 (127.9.176.196:44521)
I20240502 01:52:33.592654  9967 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.176.196:40555
I20240502 01:52:33.649995 10291 heartbeater.cc:503] Master 127.9.176.254:36089 was elected leader, sending a full tablet report...
W20240502 01:52:33.884894 10561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:33.885236 10561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:52:33.885416 10561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:33.893687 10561 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:52:33.893852 10561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:52:33.894045 10561 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.197
I20240502 01:52:33.903940 10561 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:38509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.197:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.9.176.197
--webserver_port=0
--tserver_master_addrs=127.9.176.254:36089
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.197
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:52:33.905452 10561 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:52:33.907311 10561 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:33.920194 10570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:33.920150 10567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:33.920792 10568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:33.921319 10561 server_base.cc:1034] running on GCE node
I20240502 01:52:33.921940 10561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:52:33.922946 10561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:52:33.924166 10561 hybrid_clock.cc:648] HybridClock initialized: now 1714614753924114 us; error 47 us; skew 500 ppm
I20240502 01:52:33.924506 10561 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:33.926784 10561 webserver.cc:460] Webserver started at http://127.9.176.197:33787/ using document root <none> and password file <none>
I20240502 01:52:33.927539 10561 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:33.927729 10561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:33.928074 10561 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:52:33.930796 10561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "a8c4a518929d4bb3a88bdfb1649493bd"
format_stamp: "Formatted at 2024-05-02 01:52:33 on dist-test-slave-j3hh"
I20240502 01:52:33.931622 10561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "a8c4a518929d4bb3a88bdfb1649493bd"
format_stamp: "Formatted at 2024-05-02 01:52:33 on dist-test-slave-j3hh"
I20240502 01:52:33.937191 10561 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.000s	sys 0.004s
I20240502 01:52:33.941603 10576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:52:33.943161 10561 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.000s
I20240502 01:52:33.943362 10561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "a8c4a518929d4bb3a88bdfb1649493bd"
format_stamp: "Formatted at 2024-05-02 01:52:33 on dist-test-slave-j3hh"
I20240502 01:52:33.943619 10561 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:33.957063 10561 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:52:33.957975 10561 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:52:33.958277 10561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:33.959336 10561 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:52:33.961112 10561 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:52:33.961216 10561 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:52:33.961341 10561 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:52:33.961416 10561 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:52:33.996373 10561 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.197:44061
I20240502 01:52:33.996434 10690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.197:44061 every 8 connection(s)
I20240502 01:52:33.997781 10561 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 01:52:34.002683  9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu as pid 10561
I20240502 01:52:34.002929  9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 01:52:34.006817  9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
/tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.198:0
--local_ip_for_outbound_sockets=127.9.176.198
--webserver_interface=127.9.176.198
--webserver_port=0
--tserver_master_addrs=127.9.176.254:36089
--builtin_ntp_servers=127.9.176.212:38509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:52:34.013160 10691 heartbeater.cc:347] Connected to a master server at 127.9.176.254:36089
I20240502 01:52:34.013494 10691 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:34.014331 10691 heartbeater.cc:511] Master 127.9.176.254:36089 requested a full tablet report, sending...
I20240502 01:52:34.064716  9967 ts_manager.cc:194] Registered new tserver with Master: a8c4a518929d4bb3a88bdfb1649493bd (127.9.176.197:44061)
I20240502 01:52:34.066082  9967 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.176.197:35521
I20240502 01:52:34.111953 10424 heartbeater.cc:503] Master 127.9.176.254:36089 was elected leader, sending a full tablet report...
W20240502 01:52:34.357739 10694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:34.358114 10694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:52:34.358278 10694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:34.366566 10694 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:52:34.366744 10694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:52:34.366925 10694 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.198
I20240502 01:52:34.376729 10694 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:38509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.198:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.9.176.198
--webserver_port=0
--tserver_master_addrs=127.9.176.254:36089
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.198
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:52:34.378278 10694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:52:34.380165 10694 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:34.392676 10700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:34.393165 10701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:34.393548 10703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:34.393860 10694 server_base.cc:1034] running on GCE node
I20240502 01:52:34.394480 10694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:52:34.395439 10694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:52:34.396651 10694 hybrid_clock.cc:648] HybridClock initialized: now 1714614754396592 us; error 60 us; skew 500 ppm
I20240502 01:52:34.396977 10694 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:34.399291 10694 webserver.cc:460] Webserver started at http://127.9.176.198:46003/ using document root <none> and password file <none>
I20240502 01:52:34.400067 10694 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:34.400171 10694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:34.400465 10694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:52:34.403115 10694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff"
format_stamp: "Formatted at 2024-05-02 01:52:34 on dist-test-slave-j3hh"
I20240502 01:52:34.403946 10694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff"
format_stamp: "Formatted at 2024-05-02 01:52:34 on dist-test-slave-j3hh"
I20240502 01:52:34.409333 10694 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.004s	sys 0.000s
I20240502 01:52:34.412824 10710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:52:34.414108 10694 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20240502 01:52:34.414243 10694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff"
format_stamp: "Formatted at 2024-05-02 01:52:34 on dist-test-slave-j3hh"
I20240502 01:52:34.414430 10694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:34.430575 10694 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:52:34.431284 10694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:52:34.431535 10694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:34.432600 10694 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:52:34.434100 10694 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:52:34.434201 10694 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:52:34.434316 10694 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:52:34.434392 10694 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:52:34.470476 10694 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.198:33039
I20240502 01:52:34.470547 10825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.198:33039 every 8 connection(s)
I20240502 01:52:34.471938 10694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 01:52:34.480319  9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu as pid 10694
I20240502 01:52:34.480499  9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240502 01:52:34.486142 10826 heartbeater.cc:347] Connected to a master server at 127.9.176.254:36089
I20240502 01:52:34.486495 10826 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:34.487398 10826 heartbeater.cc:511] Master 127.9.176.254:36089 requested a full tablet report, sending...
I20240502 01:52:34.532704  9967 ts_manager.cc:194] Registered new tserver with Master: fd95bcd4cdcb4cc5966050ffe2cffdff (127.9.176.198:33039)
I20240502 01:52:34.533619  9967 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.176.198:49223
I20240502 01:52:34.537185  9923 external_mini_cluster.cc:930] 6 TS(s) registered with all masters
I20240502 01:52:34.559756  9923 external_mini_cluster.cc:930] 6 TS(s) registered with all masters
I20240502 01:52:34.595667 10558 heartbeater.cc:503] Master 127.9.176.254:36089 was elected leader, sending a full tablet report...
I20240502 01:52:35.069182 10691 heartbeater.cc:503] Master 127.9.176.254:36089 was elected leader, sending a full tablet report...
I20240502 01:52:35.536886 10826 heartbeater.cc:503] Master 127.9.176.254:36089 was elected leader, sending a full tablet report...
I20240502 01:52:38.313906  9967 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56524:
name: "rebalance_test_table_0"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 01:52:38.353662 10759 tablet_service.cc:1467] Processing CreateTablet for tablet a97bdeed6d00481d8a352a67da51c7f6 (DEFAULT_TABLE table=rebalance_test_table_0 [id=6003214cc1f14993a15ed267563237ee]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:38.359474 10091 tablet_service.cc:1467] Processing CreateTablet for tablet 209e5d85bfd543b19c0c55b657bf7931 (DEFAULT_TABLE table=rebalance_test_table_0 [id=6003214cc1f14993a15ed267563237ee]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:38.360605 10092 tablet_service.cc:1467] Processing CreateTablet for tablet a97bdeed6d00481d8a352a67da51c7f6 (DEFAULT_TABLE table=rebalance_test_table_0 [id=6003214cc1f14993a15ed267563237ee]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:38.361312 10091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 209e5d85bfd543b19c0c55b657bf7931. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:38.362592 10092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a97bdeed6d00481d8a352a67da51c7f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:38.364132 10759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a97bdeed6d00481d8a352a67da51c7f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:38.364746 10225 tablet_service.cc:1467] Processing CreateTablet for tablet 75d03e64bee04a2092364ca91241cf4f (DEFAULT_TABLE table=rebalance_test_table_0 [id=6003214cc1f14993a15ed267563237ee]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:38.366446 10225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75d03e64bee04a2092364ca91241cf4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:38.376487 10357 tablet_service.cc:1467] Processing CreateTablet for tablet 209e5d85bfd543b19c0c55b657bf7931 (DEFAULT_TABLE table=rebalance_test_table_0 [id=6003214cc1f14993a15ed267563237ee]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:38.378762 10357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 209e5d85bfd543b19c0c55b657bf7931. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:38.377653 10358 tablet_service.cc:1467] Processing CreateTablet for tablet a97bdeed6d00481d8a352a67da51c7f6 (DEFAULT_TABLE table=rebalance_test_table_0 [id=6003214cc1f14993a15ed267563237ee]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:38.380159 10358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a97bdeed6d00481d8a352a67da51c7f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:38.385788 10623 tablet_service.cc:1467] Processing CreateTablet for tablet 209e5d85bfd543b19c0c55b657bf7931 (DEFAULT_TABLE table=rebalance_test_table_0 [id=6003214cc1f14993a15ed267563237ee]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:38.388088 10623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 209e5d85bfd543b19c0c55b657bf7931. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:38.386873 10624 tablet_service.cc:1467] Processing CreateTablet for tablet 75d03e64bee04a2092364ca91241cf4f (DEFAULT_TABLE table=rebalance_test_table_0 [id=6003214cc1f14993a15ed267563237ee]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:38.389233 10624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75d03e64bee04a2092364ca91241cf4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:38.388994 10492 tablet_service.cc:1467] Processing CreateTablet for tablet 75d03e64bee04a2092364ca91241cf4f (DEFAULT_TABLE table=rebalance_test_table_0 [id=6003214cc1f14993a15ed267563237ee]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:38.390892 10492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75d03e64bee04a2092364ca91241cf4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:38.419549 10852 tablet_bootstrap.cc:492] T 209e5d85bfd543b19c0c55b657bf7931 P a8c4a518929d4bb3a88bdfb1649493bd: Bootstrap starting.
I20240502 01:52:38.425529 10852 tablet_bootstrap.cc:654] T 209e5d85bfd543b19c0c55b657bf7931 P a8c4a518929d4bb3a88bdfb1649493bd: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:38.427397 10852 log.cc:824] T 209e5d85bfd543b19c0c55b657bf7931 P a8c4a518929d4bb3a88bdfb1649493bd: Log is configured to *not* fsync() on all Append() calls
I20240502 01:52:38.438059 10852 tablet_bootstrap.cc:492] T 209e5d85bfd543b19c0c55b657bf7931 P a8c4a518929d4bb3a88bdfb1649493bd: No bootstrap required, opened a new log
I20240502 01:52:38.438617 10852 ts_tablet_manager.cc:1367] T 209e5d85bfd543b19c0c55b657bf7931 P a8c4a518929d4bb3a88bdfb1649493bd: Time spent bootstrapping tablet: real 0.019s	user 0.012s	sys 0.005s
I20240502 01:52:38.441548 10853 tablet_bootstrap.cc:492] T a97bdeed6d00481d8a352a67da51c7f6 P a885359a82bd4515913f7b273591f851: Bootstrap starting.
I20240502 01:52:38.448102 10851 tablet_bootstrap.cc:492] T 75d03e64bee04a2092364ca91241cf4f P 7b7a47245f294ce9a264fcc8f6351c0f: Bootstrap starting.
I20240502 01:52:38.449116 10855 tablet_bootstrap.cc:492] T 209e5d85bfd543b19c0c55b657bf7931 P ede1dd598e3541138a19a3ea5c052bc9: Bootstrap starting.
I20240502 01:52:38.451970 10852 raft_consensus.cc:348] T 209e5d85bfd543b19c0c55b657bf7931 P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } }
I20240502 01:52:38.452698 10852 raft_consensus.cc:374] T 209e5d85bfd543b19c0c55b657bf7931 P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:38.452879 10852 raft_consensus.cc:729] T 209e5d85bfd543b19c0c55b657bf7931 P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8c4a518929d4bb3a88bdfb1649493bd, State: Initialized, Role: FOLLOWER
I20240502 01:52:38.453095 10853 tablet_bootstrap.cc:654] T a97bdeed6d00481d8a352a67da51c7f6 P a885359a82bd4515913f7b273591f851: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:38.453198 10851 tablet_bootstrap.cc:654] T 75d03e64bee04a2092364ca91241cf4f P 7b7a47245f294ce9a264fcc8f6351c0f: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:38.454037 10852 consensus_queue.cc:260] T 209e5d85bfd543b19c0c55b657bf7931 P a8c4a518929d4bb3a88bdfb1649493bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } }
I20240502 01:52:38.454589 10853 log.cc:824] T a97bdeed6d00481d8a352a67da51c7f6 P a885359a82bd4515913f7b273591f851: Log is configured to *not* fsync() on all Append() calls
I20240502 01:52:38.454883 10851 log.cc:824] T 75d03e64bee04a2092364ca91241cf4f P 7b7a47245f294ce9a264fcc8f6351c0f: Log is configured to *not* fsync() on all Append() calls
I20240502 01:52:38.456614 10850 tablet_bootstrap.cc:492] T a97bdeed6d00481d8a352a67da51c7f6 P fd95bcd4cdcb4cc5966050ffe2cffdff: Bootstrap starting.
I20240502 01:52:38.460980 10850 tablet_bootstrap.cc:654] T a97bdeed6d00481d8a352a67da51c7f6 P fd95bcd4cdcb4cc5966050ffe2cffdff: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:38.461505 10851 tablet_bootstrap.cc:492] T 75d03e64bee04a2092364ca91241cf4f P 7b7a47245f294ce9a264fcc8f6351c0f: No bootstrap required, opened a new log
I20240502 01:52:38.461539 10855 tablet_bootstrap.cc:654] T 209e5d85bfd543b19c0c55b657bf7931 P ede1dd598e3541138a19a3ea5c052bc9: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:38.461822 10851 ts_tablet_manager.cc:1367] T 75d03e64bee04a2092364ca91241cf4f P 7b7a47245f294ce9a264fcc8f6351c0f: Time spent bootstrapping tablet: real 0.014s	user 0.004s	sys 0.007s
I20240502 01:52:38.462357 10850 log.cc:824] T a97bdeed6d00481d8a352a67da51c7f6 P fd95bcd4cdcb4cc5966050ffe2cffdff: Log is configured to *not* fsync() on all Append() calls
I20240502 01:52:38.463596 10855 log.cc:824] T 209e5d85bfd543b19c0c55b657bf7931 P ede1dd598e3541138a19a3ea5c052bc9: Log is configured to *not* fsync() on all Append() calls
I20240502 01:52:38.468240 10853 tablet_bootstrap.cc:492] T a97bdeed6d00481d8a352a67da51c7f6 P a885359a82bd4515913f7b273591f851: No bootstrap required, opened a new log
I20240502 01:52:38.468693 10853 ts_tablet_manager.cc:1367] T a97bdeed6d00481d8a352a67da51c7f6 P a885359a82bd4515913f7b273591f851: Time spent bootstrapping tablet: real 0.028s	user 0.011s	sys 0.000s
I20240502 01:52:38.474586 10861 tablet_bootstrap.cc:492] T 75d03e64bee04a2092364ca91241cf4f P 34da849acb2045a9a2ef03933592dd08: Bootstrap starting.
I20240502 01:52:38.478765 10852 ts_tablet_manager.cc:1398] T 209e5d85bfd543b19c0c55b657bf7931 P a8c4a518929d4bb3a88bdfb1649493bd: Time spent starting tablet: real 0.040s	user 0.018s	sys 0.007s
I20240502 01:52:38.479400 10852 tablet_bootstrap.cc:492] T 75d03e64bee04a2092364ca91241cf4f P a8c4a518929d4bb3a88bdfb1649493bd: Bootstrap starting.
I20240502 01:52:38.481336 10852 tablet_bootstrap.cc:654] T 75d03e64bee04a2092364ca91241cf4f P a8c4a518929d4bb3a88bdfb1649493bd: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:38.484418 10851 raft_consensus.cc:348] T 75d03e64bee04a2092364ca91241cf4f P 7b7a47245f294ce9a264fcc8f6351c0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } }
I20240502 01:52:38.485028 10851 raft_consensus.cc:374] T 75d03e64bee04a2092364ca91241cf4f P 7b7a47245f294ce9a264fcc8f6351c0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:38.485157 10851 raft_consensus.cc:729] T 75d03e64bee04a2092364ca91241cf4f P 7b7a47245f294ce9a264fcc8f6351c0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b7a47245f294ce9a264fcc8f6351c0f, State: Initialized, Role: FOLLOWER
I20240502 01:52:38.486056 10851 consensus_queue.cc:260] T 75d03e64bee04a2092364ca91241cf4f P 7b7a47245f294ce9a264fcc8f6351c0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } }
I20240502 01:52:38.487001 10853 raft_consensus.cc:348] T a97bdeed6d00481d8a352a67da51c7f6 P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } }
I20240502 01:52:38.487636 10853 raft_consensus.cc:374] T a97bdeed6d00481d8a352a67da51c7f6 P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:38.487788 10853 raft_consensus.cc:729] T a97bdeed6d00481d8a352a67da51c7f6 P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a885359a82bd4515913f7b273591f851, State: Initialized, Role: FOLLOWER
I20240502 01:52:38.488034 10855 tablet_bootstrap.cc:492] T 209e5d85bfd543b19c0c55b657bf7931 P ede1dd598e3541138a19a3ea5c052bc9: No bootstrap required, opened a new log
I20240502 01:52:38.488513 10855 ts_tablet_manager.cc:1367] T 209e5d85bfd543b19c0c55b657bf7931 P ede1dd598e3541138a19a3ea5c052bc9: Time spent bootstrapping tablet: real 0.040s	user 0.011s	sys 0.004s
I20240502 01:52:38.488792 10853 consensus_queue.cc:260] T a97bdeed6d00481d8a352a67da51c7f6 P a885359a82bd4515913f7b273591f851 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } }
I20240502 01:52:38.496600 10851 ts_tablet_manager.cc:1398] T 75d03e64bee04a2092364ca91241cf4f P 7b7a47245f294ce9a264fcc8f6351c0f: Time spent starting tablet: real 0.035s	user 0.010s	sys 0.010s
I20240502 01:52:38.498589 10853 ts_tablet_manager.cc:1398] T a97bdeed6d00481d8a352a67da51c7f6 P a885359a82bd4515913f7b273591f851: Time spent starting tablet: real 0.030s	user 0.008s	sys 0.014s
I20240502 01:52:38.499146 10853 tablet_bootstrap.cc:492] T 209e5d85bfd543b19c0c55b657bf7931 P a885359a82bd4515913f7b273591f851: Bootstrap starting.
I20240502 01:52:38.501417 10853 tablet_bootstrap.cc:654] T 209e5d85bfd543b19c0c55b657bf7931 P a885359a82bd4515913f7b273591f851: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:38.502467 10850 tablet_bootstrap.cc:492] T a97bdeed6d00481d8a352a67da51c7f6 P fd95bcd4cdcb4cc5966050ffe2cffdff: No bootstrap required, opened a new log
I20240502 01:52:38.502799 10850 ts_tablet_manager.cc:1367] T a97bdeed6d00481d8a352a67da51c7f6 P fd95bcd4cdcb4cc5966050ffe2cffdff: Time spent bootstrapping tablet: real 0.047s	user 0.003s	sys 0.017s
W20240502 01:52:38.503396 10692 tablet.cc:2369] T 209e5d85bfd543b19c0c55b657bf7931 P a8c4a518929d4bb3a88bdfb1649493bd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:52:38.508942 10861 tablet_bootstrap.cc:654] T 75d03e64bee04a2092364ca91241cf4f P 34da849acb2045a9a2ef03933592dd08: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:38.510308 10861 log.cc:824] T 75d03e64bee04a2092364ca91241cf4f P 34da849acb2045a9a2ef03933592dd08: Log is configured to *not* fsync() on all Append() calls
I20240502 01:52:38.499230 10855 raft_consensus.cc:348] T 209e5d85bfd543b19c0c55b657bf7931 P ede1dd598e3541138a19a3ea5c052bc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } }
I20240502 01:52:38.511116 10855 raft_consensus.cc:374] T 209e5d85bfd543b19c0c55b657bf7931 P ede1dd598e3541138a19a3ea5c052bc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:38.511233 10855 raft_consensus.cc:729] T 209e5d85bfd543b19c0c55b657bf7931 P ede1dd598e3541138a19a3ea5c052bc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ede1dd598e3541138a19a3ea5c052bc9, State: Initialized, Role: FOLLOWER
I20240502 01:52:38.512045 10855 consensus_queue.cc:260] T 209e5d85bfd543b19c0c55b657bf7931 P ede1dd598e3541138a19a3ea5c052bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } }
I20240502 01:52:38.513062 10850 raft_consensus.cc:348] T a97bdeed6d00481d8a352a67da51c7f6 P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } }
I20240502 01:52:38.513513 10850 raft_consensus.cc:374] T a97bdeed6d00481d8a352a67da51c7f6 P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:38.513660 10850 raft_consensus.cc:729] T a97bdeed6d00481d8a352a67da51c7f6 P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd95bcd4cdcb4cc5966050ffe2cffdff, State: Initialized, Role: FOLLOWER
I20240502 01:52:38.514391 10852 tablet_bootstrap.cc:492] T 75d03e64bee04a2092364ca91241cf4f P a8c4a518929d4bb3a88bdfb1649493bd: No bootstrap required, opened a new log
I20240502 01:52:38.514329 10850 consensus_queue.cc:260] T a97bdeed6d00481d8a352a67da51c7f6 P fd95bcd4cdcb4cc5966050ffe2cffdff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } }
I20240502 01:52:38.514581 10852 ts_tablet_manager.cc:1367] T 75d03e64bee04a2092364ca91241cf4f P a8c4a518929d4bb3a88bdfb1649493bd: Time spent bootstrapping tablet: real 0.035s	user 0.012s	sys 0.016s
I20240502 01:52:38.515791 10855 ts_tablet_manager.cc:1398] T 209e5d85bfd543b19c0c55b657bf7931 P ede1dd598e3541138a19a3ea5c052bc9: Time spent starting tablet: real 0.027s	user 0.014s	sys 0.000s
I20240502 01:52:38.515753 10852 raft_consensus.cc:348] T 75d03e64bee04a2092364ca91241cf4f P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } }
I20240502 01:52:38.516000 10852 raft_consensus.cc:374] T 75d03e64bee04a2092364ca91241cf4f P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:38.516155 10852 raft_consensus.cc:729] T 75d03e64bee04a2092364ca91241cf4f P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8c4a518929d4bb3a88bdfb1649493bd, State: Initialized, Role: FOLLOWER
I20240502 01:52:38.516422 10855 tablet_bootstrap.cc:492] T a97bdeed6d00481d8a352a67da51c7f6 P ede1dd598e3541138a19a3ea5c052bc9: Bootstrap starting.
I20240502 01:52:38.516410 10852 consensus_queue.cc:260] T 75d03e64bee04a2092364ca91241cf4f P a8c4a518929d4bb3a88bdfb1649493bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } }
I20240502 01:52:38.516904 10853 tablet_bootstrap.cc:492] T 209e5d85bfd543b19c0c55b657bf7931 P a885359a82bd4515913f7b273591f851: No bootstrap required, opened a new log
I20240502 01:52:38.517093 10853 ts_tablet_manager.cc:1367] T 209e5d85bfd543b19c0c55b657bf7931 P a885359a82bd4515913f7b273591f851: Time spent bootstrapping tablet: real 0.018s	user 0.007s	sys 0.007s
I20240502 01:52:38.517215 10852 ts_tablet_manager.cc:1398] T 75d03e64bee04a2092364ca91241cf4f P a8c4a518929d4bb3a88bdfb1649493bd: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.003s
I20240502 01:52:38.518602 10855 tablet_bootstrap.cc:654] T a97bdeed6d00481d8a352a67da51c7f6 P ede1dd598e3541138a19a3ea5c052bc9: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:38.519345 10853 raft_consensus.cc:348] T 209e5d85bfd543b19c0c55b657bf7931 P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } }
I20240502 01:52:38.519515 10853 raft_consensus.cc:374] T 209e5d85bfd543b19c0c55b657bf7931 P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:38.519630 10853 raft_consensus.cc:729] T 209e5d85bfd543b19c0c55b657bf7931 P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a885359a82bd4515913f7b273591f851, State: Initialized, Role: FOLLOWER
I20240502 01:52:38.519889 10853 consensus_queue.cc:260] T 209e5d85bfd543b19c0c55b657bf7931 P a885359a82bd4515913f7b273591f851 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } }
I20240502 01:52:38.520336 10853 ts_tablet_manager.cc:1398] T 209e5d85bfd543b19c0c55b657bf7931 P a885359a82bd4515913f7b273591f851: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 01:52:38.521322 10861 tablet_bootstrap.cc:492] T 75d03e64bee04a2092364ca91241cf4f P 34da849acb2045a9a2ef03933592dd08: No bootstrap required, opened a new log
I20240502 01:52:38.521628 10861 ts_tablet_manager.cc:1367] T 75d03e64bee04a2092364ca91241cf4f P 34da849acb2045a9a2ef03933592dd08: Time spent bootstrapping tablet: real 0.047s	user 0.008s	sys 0.000s
I20240502 01:52:38.523792 10856 raft_consensus.cc:482] T 75d03e64bee04a2092364ca91241cf4f P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:38.524006 10856 raft_consensus.cc:504] T 75d03e64bee04a2092364ca91241cf4f P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } }
I20240502 01:52:38.525993 10856 leader_election.cc:288] T 75d03e64bee04a2092364ca91241cf4f P a8c4a518929d4bb3a88bdfb1649493bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b7a47245f294ce9a264fcc8f6351c0f (127.9.176.194:43645), 34da849acb2045a9a2ef03933592dd08 (127.9.176.196:44521)
I20240502 01:52:38.528884 10850 ts_tablet_manager.cc:1398] T a97bdeed6d00481d8a352a67da51c7f6 P fd95bcd4cdcb4cc5966050ffe2cffdff: Time spent starting tablet: real 0.026s	user 0.004s	sys 0.008s
I20240502 01:52:38.531052 10861 raft_consensus.cc:348] T 75d03e64bee04a2092364ca91241cf4f P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } }
I20240502 01:52:38.531539 10861 raft_consensus.cc:374] T 75d03e64bee04a2092364ca91241cf4f P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:38.531674 10861 raft_consensus.cc:729] T 75d03e64bee04a2092364ca91241cf4f P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34da849acb2045a9a2ef03933592dd08, State: Initialized, Role: FOLLOWER
I20240502 01:52:38.532465 10861 consensus_queue.cc:260] T 75d03e64bee04a2092364ca91241cf4f P 34da849acb2045a9a2ef03933592dd08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } }
I20240502 01:52:38.534193 10855 tablet_bootstrap.cc:492] T a97bdeed6d00481d8a352a67da51c7f6 P ede1dd598e3541138a19a3ea5c052bc9: No bootstrap required, opened a new log
I20240502 01:52:38.534365 10855 ts_tablet_manager.cc:1367] T a97bdeed6d00481d8a352a67da51c7f6 P ede1dd598e3541138a19a3ea5c052bc9: Time spent bootstrapping tablet: real 0.018s	user 0.006s	sys 0.000s
I20240502 01:52:38.535519 10855 raft_consensus.cc:348] T a97bdeed6d00481d8a352a67da51c7f6 P ede1dd598e3541138a19a3ea5c052bc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } }
I20240502 01:52:38.535785 10855 raft_consensus.cc:374] T a97bdeed6d00481d8a352a67da51c7f6 P ede1dd598e3541138a19a3ea5c052bc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:38.535877 10855 raft_consensus.cc:729] T a97bdeed6d00481d8a352a67da51c7f6 P ede1dd598e3541138a19a3ea5c052bc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ede1dd598e3541138a19a3ea5c052bc9, State: Initialized, Role: FOLLOWER
I20240502 01:52:38.536190 10855 consensus_queue.cc:260] T a97bdeed6d00481d8a352a67da51c7f6 P ede1dd598e3541138a19a3ea5c052bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } }
I20240502 01:52:38.536929 10855 ts_tablet_manager.cc:1398] T a97bdeed6d00481d8a352a67da51c7f6 P ede1dd598e3541138a19a3ea5c052bc9: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.000s
I20240502 01:52:38.545238 10861 ts_tablet_manager.cc:1398] T 75d03e64bee04a2092364ca91241cf4f P 34da849acb2045a9a2ef03933592dd08: Time spent starting tablet: real 0.023s	user 0.007s	sys 0.016s
I20240502 01:52:38.546545 10512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75d03e64bee04a2092364ca91241cf4f" candidate_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34da849acb2045a9a2ef03933592dd08" is_pre_election: true
I20240502 01:52:38.547235 10512 raft_consensus.cc:2454] T 75d03e64bee04a2092364ca91241cf4f P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8c4a518929d4bb3a88bdfb1649493bd in term 0.
I20240502 01:52:38.548100 10577 leader_election.cc:302] T 75d03e64bee04a2092364ca91241cf4f P a8c4a518929d4bb3a88bdfb1649493bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34da849acb2045a9a2ef03933592dd08, a8c4a518929d4bb3a88bdfb1649493bd; no voters: 
I20240502 01:52:38.548676 10856 raft_consensus.cc:2781] T 75d03e64bee04a2092364ca91241cf4f P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:38.548879 10856 raft_consensus.cc:482] T 75d03e64bee04a2092364ca91241cf4f P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:38.548990 10856 raft_consensus.cc:3037] T 75d03e64bee04a2092364ca91241cf4f P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:38.551702 10856 raft_consensus.cc:504] T 75d03e64bee04a2092364ca91241cf4f P a8c4a518929d4bb3a88bdfb1649493bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } }
I20240502 01:52:38.551754 10246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75d03e64bee04a2092364ca91241cf4f" candidate_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" is_pre_election: true
I20240502 01:52:38.552315 10246 raft_consensus.cc:2454] T 75d03e64bee04a2092364ca91241cf4f P 7b7a47245f294ce9a264fcc8f6351c0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8c4a518929d4bb3a88bdfb1649493bd in term 0.
I20240502 01:52:38.552407 10856 leader_election.cc:288] T 75d03e64bee04a2092364ca91241cf4f P a8c4a518929d4bb3a88bdfb1649493bd [CANDIDATE]: Term 1 election: Requested vote from peers 7b7a47245f294ce9a264fcc8f6351c0f (127.9.176.194:43645), 34da849acb2045a9a2ef03933592dd08 (127.9.176.196:44521)
I20240502 01:52:38.552920 10246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75d03e64bee04a2092364ca91241cf4f" candidate_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b7a47245f294ce9a264fcc8f6351c0f"
I20240502 01:52:38.553066 10512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75d03e64bee04a2092364ca91241cf4f" candidate_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34da849acb2045a9a2ef03933592dd08"
I20240502 01:52:38.553130 10246 raft_consensus.cc:3037] T 75d03e64bee04a2092364ca91241cf4f P 7b7a47245f294ce9a264fcc8f6351c0f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:38.553275 10512 raft_consensus.cc:3037] T 75d03e64bee04a2092364ca91241cf4f P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Advancing to term 1
W20240502 01:52:38.554366 10425 tablet.cc:2369] T 209e5d85bfd543b19c0c55b657bf7931 P ede1dd598e3541138a19a3ea5c052bc9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:52:38.555289 10246 raft_consensus.cc:2454] T 75d03e64bee04a2092364ca91241cf4f P 7b7a47245f294ce9a264fcc8f6351c0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8c4a518929d4bb3a88bdfb1649493bd in term 1.
I20240502 01:52:38.555822 10577 leader_election.cc:302] T 75d03e64bee04a2092364ca91241cf4f P a8c4a518929d4bb3a88bdfb1649493bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b7a47245f294ce9a264fcc8f6351c0f, a8c4a518929d4bb3a88bdfb1649493bd; no voters: 
I20240502 01:52:38.556151 10856 raft_consensus.cc:2781] T 75d03e64bee04a2092364ca91241cf4f P a8c4a518929d4bb3a88bdfb1649493bd [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:38.556504 10512 raft_consensus.cc:2454] T 75d03e64bee04a2092364ca91241cf4f P 34da849acb2045a9a2ef03933592dd08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8c4a518929d4bb3a88bdfb1649493bd in term 1.
I20240502 01:52:38.556777 10856 raft_consensus.cc:686] T 75d03e64bee04a2092364ca91241cf4f P a8c4a518929d4bb3a88bdfb1649493bd [term 1 LEADER]: Becoming Leader. State: Replica: a8c4a518929d4bb3a88bdfb1649493bd, State: Running, Role: LEADER
I20240502 01:52:38.557366 10856 consensus_queue.cc:237] T 75d03e64bee04a2092364ca91241cf4f P a8c4a518929d4bb3a88bdfb1649493bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } }
I20240502 01:52:38.562618  9967 catalog_manager.cc:5484] T 75d03e64bee04a2092364ca91241cf4f P a8c4a518929d4bb3a88bdfb1649493bd reported cstate change: term changed from 0 to 1, leader changed from <none> to a8c4a518929d4bb3a88bdfb1649493bd (127.9.176.197). New cstate: current_term: 1 leader_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } health_report { overall_health: UNKNOWN } } }
W20240502 01:52:38.583086 10292 tablet.cc:2369] T 75d03e64bee04a2092364ca91241cf4f P 7b7a47245f294ce9a264fcc8f6351c0f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 01:52:38.609408 10158 tablet.cc:2369] T 209e5d85bfd543b19c0c55b657bf7931 P a885359a82bd4515913f7b273591f851: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:52:38.631371 10864 raft_consensus.cc:482] T 209e5d85bfd543b19c0c55b657bf7931 P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:38.631666 10864 raft_consensus.cc:504] T 209e5d85bfd543b19c0c55b657bf7931 P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } }
I20240502 01:52:38.633423 10864 leader_election.cc:288] T 209e5d85bfd543b19c0c55b657bf7931 P a885359a82bd4515913f7b273591f851 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ede1dd598e3541138a19a3ea5c052bc9 (127.9.176.195:38701), a8c4a518929d4bb3a88bdfb1649493bd (127.9.176.197:44061)
I20240502 01:52:38.641417 10644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "209e5d85bfd543b19c0c55b657bf7931" candidate_uuid: "a885359a82bd4515913f7b273591f851" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" is_pre_election: true
I20240502 01:52:38.641887 10644 raft_consensus.cc:2454] T 209e5d85bfd543b19c0c55b657bf7931 P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a885359a82bd4515913f7b273591f851 in term 0.
I20240502 01:52:38.641769 10378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "209e5d85bfd543b19c0c55b657bf7931" candidate_uuid: "a885359a82bd4515913f7b273591f851" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ede1dd598e3541138a19a3ea5c052bc9" is_pre_election: true
I20240502 01:52:38.642396 10378 raft_consensus.cc:2454] T 209e5d85bfd543b19c0c55b657bf7931 P ede1dd598e3541138a19a3ea5c052bc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a885359a82bd4515913f7b273591f851 in term 0.
I20240502 01:52:38.642890 10046 leader_election.cc:302] T 209e5d85bfd543b19c0c55b657bf7931 P a885359a82bd4515913f7b273591f851 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a885359a82bd4515913f7b273591f851, a8c4a518929d4bb3a88bdfb1649493bd; no voters: 
I20240502 01:52:38.643373 10864 raft_consensus.cc:2781] T 209e5d85bfd543b19c0c55b657bf7931 P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:38.643518 10864 raft_consensus.cc:482] T 209e5d85bfd543b19c0c55b657bf7931 P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:38.643661 10864 raft_consensus.cc:3037] T 209e5d85bfd543b19c0c55b657bf7931 P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:38.645797 10864 raft_consensus.cc:504] T 209e5d85bfd543b19c0c55b657bf7931 P a885359a82bd4515913f7b273591f851 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } }
I20240502 01:52:38.646517 10864 leader_election.cc:288] T 209e5d85bfd543b19c0c55b657bf7931 P a885359a82bd4515913f7b273591f851 [CANDIDATE]: Term 1 election: Requested vote from peers ede1dd598e3541138a19a3ea5c052bc9 (127.9.176.195:38701), a8c4a518929d4bb3a88bdfb1649493bd (127.9.176.197:44061)
I20240502 01:52:38.647048 10378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "209e5d85bfd543b19c0c55b657bf7931" candidate_uuid: "a885359a82bd4515913f7b273591f851" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ede1dd598e3541138a19a3ea5c052bc9"
I20240502 01:52:38.647161 10644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "209e5d85bfd543b19c0c55b657bf7931" candidate_uuid: "a885359a82bd4515913f7b273591f851" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8c4a518929d4bb3a88bdfb1649493bd"
I20240502 01:52:38.647259 10378 raft_consensus.cc:3037] T 209e5d85bfd543b19c0c55b657bf7931 P ede1dd598e3541138a19a3ea5c052bc9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:38.647380 10644 raft_consensus.cc:3037] T 209e5d85bfd543b19c0c55b657bf7931 P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:38.649816 10378 raft_consensus.cc:2454] T 209e5d85bfd543b19c0c55b657bf7931 P ede1dd598e3541138a19a3ea5c052bc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a885359a82bd4515913f7b273591f851 in term 1.
I20240502 01:52:38.649827 10644 raft_consensus.cc:2454] T 209e5d85bfd543b19c0c55b657bf7931 P a8c4a518929d4bb3a88bdfb1649493bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a885359a82bd4515913f7b273591f851 in term 1.
I20240502 01:52:38.650355 10046 leader_election.cc:302] T 209e5d85bfd543b19c0c55b657bf7931 P a885359a82bd4515913f7b273591f851 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a885359a82bd4515913f7b273591f851, a8c4a518929d4bb3a88bdfb1649493bd; no voters: 
I20240502 01:52:38.650712 10864 raft_consensus.cc:2781] T 209e5d85bfd543b19c0c55b657bf7931 P a885359a82bd4515913f7b273591f851 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:38.651165 10864 raft_consensus.cc:686] T 209e5d85bfd543b19c0c55b657bf7931 P a885359a82bd4515913f7b273591f851 [term 1 LEADER]: Becoming Leader. State: Replica: a885359a82bd4515913f7b273591f851, State: Running, Role: LEADER
I20240502 01:52:38.651775 10864 consensus_queue.cc:237] T 209e5d85bfd543b19c0c55b657bf7931 P a885359a82bd4515913f7b273591f851 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } }
I20240502 01:52:38.656544  9967 catalog_manager.cc:5484] T 209e5d85bfd543b19c0c55b657bf7931 P a885359a82bd4515913f7b273591f851 reported cstate change: term changed from 0 to 1, leader changed from <none> to a885359a82bd4515913f7b273591f851 (127.9.176.193). New cstate: current_term: 1 leader_uuid: "a885359a82bd4515913f7b273591f851" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } health_report { overall_health: UNKNOWN } } }
I20240502 01:52:38.667138 10868 raft_consensus.cc:482] T a97bdeed6d00481d8a352a67da51c7f6 P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:38.667459 10868 raft_consensus.cc:504] T a97bdeed6d00481d8a352a67da51c7f6 P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } }
I20240502 01:52:38.669389 10868 leader_election.cc:288] T a97bdeed6d00481d8a352a67da51c7f6 P fd95bcd4cdcb4cc5966050ffe2cffdff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ede1dd598e3541138a19a3ea5c052bc9 (127.9.176.195:38701), a885359a82bd4515913f7b273591f851 (127.9.176.193:38373)
I20240502 01:52:38.676764 10378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a97bdeed6d00481d8a352a67da51c7f6" candidate_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ede1dd598e3541138a19a3ea5c052bc9" is_pre_election: true
I20240502 01:52:38.677134 10378 raft_consensus.cc:2454] T a97bdeed6d00481d8a352a67da51c7f6 P ede1dd598e3541138a19a3ea5c052bc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd95bcd4cdcb4cc5966050ffe2cffdff in term 0.
I20240502 01:52:38.677112 10112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a97bdeed6d00481d8a352a67da51c7f6" candidate_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a885359a82bd4515913f7b273591f851" is_pre_election: true
I20240502 01:52:38.677567 10112 raft_consensus.cc:2454] T a97bdeed6d00481d8a352a67da51c7f6 P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd95bcd4cdcb4cc5966050ffe2cffdff in term 0.
I20240502 01:52:38.677726 10713 leader_election.cc:302] T a97bdeed6d00481d8a352a67da51c7f6 P fd95bcd4cdcb4cc5966050ffe2cffdff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ede1dd598e3541138a19a3ea5c052bc9, fd95bcd4cdcb4cc5966050ffe2cffdff; no voters: 
I20240502 01:52:38.678169 10868 raft_consensus.cc:2781] T a97bdeed6d00481d8a352a67da51c7f6 P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:38.678308 10868 raft_consensus.cc:482] T a97bdeed6d00481d8a352a67da51c7f6 P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:38.678414 10868 raft_consensus.cc:3037] T a97bdeed6d00481d8a352a67da51c7f6 P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:38.680637 10868 raft_consensus.cc:504] T a97bdeed6d00481d8a352a67da51c7f6 P fd95bcd4cdcb4cc5966050ffe2cffdff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } }
I20240502 01:52:38.681320 10868 leader_election.cc:288] T a97bdeed6d00481d8a352a67da51c7f6 P fd95bcd4cdcb4cc5966050ffe2cffdff [CANDIDATE]: Term 1 election: Requested vote from peers ede1dd598e3541138a19a3ea5c052bc9 (127.9.176.195:38701), a885359a82bd4515913f7b273591f851 (127.9.176.193:38373)
I20240502 01:52:38.681839 10112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a97bdeed6d00481d8a352a67da51c7f6" candidate_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a885359a82bd4515913f7b273591f851"
I20240502 01:52:38.681846 10378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a97bdeed6d00481d8a352a67da51c7f6" candidate_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ede1dd598e3541138a19a3ea5c052bc9"
I20240502 01:52:38.682020 10112 raft_consensus.cc:3037] T a97bdeed6d00481d8a352a67da51c7f6 P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:38.682072 10378 raft_consensus.cc:3037] T a97bdeed6d00481d8a352a67da51c7f6 P ede1dd598e3541138a19a3ea5c052bc9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:38.684199 10378 raft_consensus.cc:2454] T a97bdeed6d00481d8a352a67da51c7f6 P ede1dd598e3541138a19a3ea5c052bc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd95bcd4cdcb4cc5966050ffe2cffdff in term 1.
I20240502 01:52:38.684242 10112 raft_consensus.cc:2454] T a97bdeed6d00481d8a352a67da51c7f6 P a885359a82bd4515913f7b273591f851 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd95bcd4cdcb4cc5966050ffe2cffdff in term 1.
I20240502 01:52:38.684722 10713 leader_election.cc:302] T a97bdeed6d00481d8a352a67da51c7f6 P fd95bcd4cdcb4cc5966050ffe2cffdff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ede1dd598e3541138a19a3ea5c052bc9, fd95bcd4cdcb4cc5966050ffe2cffdff; no voters: 
I20240502 01:52:38.685110 10868 raft_consensus.cc:2781] T a97bdeed6d00481d8a352a67da51c7f6 P fd95bcd4cdcb4cc5966050ffe2cffdff [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:38.685654 10868 raft_consensus.cc:686] T a97bdeed6d00481d8a352a67da51c7f6 P fd95bcd4cdcb4cc5966050ffe2cffdff [term 1 LEADER]: Becoming Leader. State: Replica: fd95bcd4cdcb4cc5966050ffe2cffdff, State: Running, Role: LEADER
I20240502 01:52:38.686159 10868 consensus_queue.cc:237] T a97bdeed6d00481d8a352a67da51c7f6 P fd95bcd4cdcb4cc5966050ffe2cffdff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } }
I20240502 01:52:38.691370  9967 catalog_manager.cc:5484] T a97bdeed6d00481d8a352a67da51c7f6 P fd95bcd4cdcb4cc5966050ffe2cffdff reported cstate change: term changed from 0 to 1, leader changed from <none> to fd95bcd4cdcb4cc5966050ffe2cffdff (127.9.176.198). New cstate: current_term: 1 leader_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } health_report { overall_health: UNKNOWN } } }
W20240502 01:52:38.727552 10827 tablet.cc:2369] T a97bdeed6d00481d8a352a67da51c7f6 P fd95bcd4cdcb4cc5966050ffe2cffdff: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 01:52:38.778822 10559 tablet.cc:2369] T 75d03e64bee04a2092364ca91241cf4f P 34da849acb2045a9a2ef03933592dd08: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 01:52:39.064661 10883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:39.065052 10883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:39.072942 10883 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 01:52:39.105670 10894 consensus_queue.cc:1035] T 75d03e64bee04a2092364ca91241cf4f P a8c4a518929d4bb3a88bdfb1649493bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_0'
I20240502 01:52:39.119421 10894 consensus_queue.cc:1035] T 75d03e64bee04a2092364ca91241cf4f P a8c4a518929d4bb3a88bdfb1649493bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:39.140075 10864 consensus_queue.cc:1035] T 209e5d85bfd543b19c0c55b657bf7931 P a885359a82bd4515913f7b273591f851 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:39.149636 10864 consensus_queue.cc:1035] T 209e5d85bfd543b19c0c55b657bf7931 P a885359a82bd4515913f7b273591f851 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:39.156095 10868 consensus_queue.cc:1035] T a97bdeed6d00481d8a352a67da51c7f6 P fd95bcd4cdcb4cc5966050ffe2cffdff [LEADER]: Connected to new peer: Peer: permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:39.166350 10868 consensus_queue.cc:1035] T a97bdeed6d00481d8a352a67da51c7f6 P fd95bcd4cdcb4cc5966050ffe2cffdff [LEADER]: Connected to new peer: Peer: permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 249.646 ms
  time per row: 0.124823 ms
I20240502 01:52:39.485564  9963 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56524:
name: "rebalance_test_table_1"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 01:52:39.494818 10492 tablet_service.cc:1467] Processing CreateTablet for tablet ec9d18ab22e1466c8daa6b554ccaeebb (DEFAULT_TABLE table=rebalance_test_table_1 [id=2b8216e90e9c46c9a93a92cbdb44b214]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:39.495358 10492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec9d18ab22e1466c8daa6b554ccaeebb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:39.495273 10759 tablet_service.cc:1467] Processing CreateTablet for tablet ec9d18ab22e1466c8daa6b554ccaeebb (DEFAULT_TABLE table=rebalance_test_table_1 [id=2b8216e90e9c46c9a93a92cbdb44b214]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:39.495827 10759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec9d18ab22e1466c8daa6b554ccaeebb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:39.495867 10758 tablet_service.cc:1467] Processing CreateTablet for tablet a92d5a681b364e50896bdda0d92bb10a (DEFAULT_TABLE table=rebalance_test_table_1 [id=2b8216e90e9c46c9a93a92cbdb44b214]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:39.496022 10623 tablet_service.cc:1467] Processing CreateTablet for tablet 8fcb99ef99114ca59130ccef6d0803b3 (DEFAULT_TABLE table=rebalance_test_table_1 [id=2b8216e90e9c46c9a93a92cbdb44b214]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:39.496440 10623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fcb99ef99114ca59130ccef6d0803b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:39.496454 10758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a92d5a681b364e50896bdda0d92bb10a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:39.503001 10225 tablet_service.cc:1467] Processing CreateTablet for tablet ec9d18ab22e1466c8daa6b554ccaeebb (DEFAULT_TABLE table=rebalance_test_table_1 [id=2b8216e90e9c46c9a93a92cbdb44b214]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:39.503021 10224 tablet_service.cc:1467] Processing CreateTablet for tablet a92d5a681b364e50896bdda0d92bb10a (DEFAULT_TABLE table=rebalance_test_table_1 [id=2b8216e90e9c46c9a93a92cbdb44b214]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:39.503698 10224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a92d5a681b364e50896bdda0d92bb10a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:39.503544 10491 tablet_service.cc:1467] Processing CreateTablet for tablet a92d5a681b364e50896bdda0d92bb10a (DEFAULT_TABLE table=rebalance_test_table_1 [id=2b8216e90e9c46c9a93a92cbdb44b214]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:39.504138 10491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a92d5a681b364e50896bdda0d92bb10a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:39.504282 10092 tablet_service.cc:1467] Processing CreateTablet for tablet 8fcb99ef99114ca59130ccef6d0803b3 (DEFAULT_TABLE table=rebalance_test_table_1 [id=2b8216e90e9c46c9a93a92cbdb44b214]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:39.504707 10092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fcb99ef99114ca59130ccef6d0803b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:39.504660 10358 tablet_service.cc:1467] Processing CreateTablet for tablet 8fcb99ef99114ca59130ccef6d0803b3 (DEFAULT_TABLE table=rebalance_test_table_1 [id=2b8216e90e9c46c9a93a92cbdb44b214]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:39.505115 10358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fcb99ef99114ca59130ccef6d0803b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:39.510320 10225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec9d18ab22e1466c8daa6b554ccaeebb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:39.517719 10965 tablet_bootstrap.cc:492] T 8fcb99ef99114ca59130ccef6d0803b3 P a885359a82bd4515913f7b273591f851: Bootstrap starting.
I20240502 01:52:39.518537 10966 tablet_bootstrap.cc:492] T 8fcb99ef99114ca59130ccef6d0803b3 P ede1dd598e3541138a19a3ea5c052bc9: Bootstrap starting.
I20240502 01:52:39.518620 10962 tablet_bootstrap.cc:492] T ec9d18ab22e1466c8daa6b554ccaeebb P 34da849acb2045a9a2ef03933592dd08: Bootstrap starting.
I20240502 01:52:39.520244 10967 tablet_bootstrap.cc:492] T ec9d18ab22e1466c8daa6b554ccaeebb P fd95bcd4cdcb4cc5966050ffe2cffdff: Bootstrap starting.
I20240502 01:52:39.520421 10964 tablet_bootstrap.cc:492] T a92d5a681b364e50896bdda0d92bb10a P 7b7a47245f294ce9a264fcc8f6351c0f: Bootstrap starting.
I20240502 01:52:39.521454 10965 tablet_bootstrap.cc:654] T 8fcb99ef99114ca59130ccef6d0803b3 P a885359a82bd4515913f7b273591f851: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:39.522228 10966 tablet_bootstrap.cc:654] T 8fcb99ef99114ca59130ccef6d0803b3 P ede1dd598e3541138a19a3ea5c052bc9: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:39.522828 10967 tablet_bootstrap.cc:654] T ec9d18ab22e1466c8daa6b554ccaeebb P fd95bcd4cdcb4cc5966050ffe2cffdff: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:39.523648 10963 tablet_bootstrap.cc:492] T 8fcb99ef99114ca59130ccef6d0803b3 P a8c4a518929d4bb3a88bdfb1649493bd: Bootstrap starting.
I20240502 01:52:39.526060 10963 tablet_bootstrap.cc:654] T 8fcb99ef99114ca59130ccef6d0803b3 P a8c4a518929d4bb3a88bdfb1649493bd: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:39.529165 10964 tablet_bootstrap.cc:654] T a92d5a681b364e50896bdda0d92bb10a P 7b7a47245f294ce9a264fcc8f6351c0f: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:39.531646 10967 tablet_bootstrap.cc:492] T ec9d18ab22e1466c8daa6b554ccaeebb P fd95bcd4cdcb4cc5966050ffe2cffdff: No bootstrap required, opened a new log
I20240502 01:52:39.531816 10967 ts_tablet_manager.cc:1367] T ec9d18ab22e1466c8daa6b554ccaeebb P fd95bcd4cdcb4cc5966050ffe2cffdff: Time spent bootstrapping tablet: real 0.012s	user 0.005s	sys 0.005s
I20240502 01:52:39.533002 10967 raft_consensus.cc:348] T ec9d18ab22e1466c8daa6b554ccaeebb P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } } peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } }
I20240502 01:52:39.533314 10967 raft_consensus.cc:374] T ec9d18ab22e1466c8daa6b554ccaeebb P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:39.533408 10967 raft_consensus.cc:729] T ec9d18ab22e1466c8daa6b554ccaeebb P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd95bcd4cdcb4cc5966050ffe2cffdff, State: Initialized, Role: FOLLOWER
I20240502 01:52:39.533685 10967 consensus_queue.cc:260] T ec9d18ab22e1466c8daa6b554ccaeebb P fd95bcd4cdcb4cc5966050ffe2cffdff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } } peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } }
I20240502 01:52:39.534518 10967 ts_tablet_manager.cc:1398] T ec9d18ab22e1466c8daa6b554ccaeebb P fd95bcd4cdcb4cc5966050ffe2cffdff: Time spent starting tablet: real 0.003s	user 0.003s	sys 0.000s
I20240502 01:52:39.535220 10967 tablet_bootstrap.cc:492] T a92d5a681b364e50896bdda0d92bb10a P fd95bcd4cdcb4cc5966050ffe2cffdff: Bootstrap starting.
I20240502 01:52:39.537092 10967 tablet_bootstrap.cc:654] T a92d5a681b364e50896bdda0d92bb10a P fd95bcd4cdcb4cc5966050ffe2cffdff: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:39.537696 10962 tablet_bootstrap.cc:654] T ec9d18ab22e1466c8daa6b554ccaeebb P 34da849acb2045a9a2ef03933592dd08: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:39.545035 10963 tablet_bootstrap.cc:492] T 8fcb99ef99114ca59130ccef6d0803b3 P a8c4a518929d4bb3a88bdfb1649493bd: No bootstrap required, opened a new log
I20240502 01:52:39.545285 10963 ts_tablet_manager.cc:1367] T 8fcb99ef99114ca59130ccef6d0803b3 P a8c4a518929d4bb3a88bdfb1649493bd: Time spent bootstrapping tablet: real 0.022s	user 0.003s	sys 0.003s
I20240502 01:52:39.547998 10964 tablet_bootstrap.cc:492] T a92d5a681b364e50896bdda0d92bb10a P 7b7a47245f294ce9a264fcc8f6351c0f: No bootstrap required, opened a new log
I20240502 01:52:39.548223 10964 ts_tablet_manager.cc:1367] T a92d5a681b364e50896bdda0d92bb10a P 7b7a47245f294ce9a264fcc8f6351c0f: Time spent bootstrapping tablet: real 0.028s	user 0.006s	sys 0.001s
I20240502 01:52:39.549572 10964 raft_consensus.cc:348] T a92d5a681b364e50896bdda0d92bb10a P 7b7a47245f294ce9a264fcc8f6351c0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } } peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } }
I20240502 01:52:39.551985 10964 raft_consensus.cc:374] T a92d5a681b364e50896bdda0d92bb10a P 7b7a47245f294ce9a264fcc8f6351c0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:39.552161 10964 raft_consensus.cc:729] T a92d5a681b364e50896bdda0d92bb10a P 7b7a47245f294ce9a264fcc8f6351c0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b7a47245f294ce9a264fcc8f6351c0f, State: Initialized, Role: FOLLOWER
I20240502 01:52:39.552556 10964 consensus_queue.cc:260] T a92d5a681b364e50896bdda0d92bb10a P 7b7a47245f294ce9a264fcc8f6351c0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } } peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } }
I20240502 01:52:39.553406 10964 ts_tablet_manager.cc:1398] T a92d5a681b364e50896bdda0d92bb10a P 7b7a47245f294ce9a264fcc8f6351c0f: Time spent starting tablet: real 0.005s	user 0.000s	sys 0.003s
I20240502 01:52:39.556032 10963 raft_consensus.cc:348] T 8fcb99ef99114ca59130ccef6d0803b3 P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } }
I20240502 01:52:39.556370 10963 raft_consensus.cc:374] T 8fcb99ef99114ca59130ccef6d0803b3 P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:39.556478 10963 raft_consensus.cc:729] T 8fcb99ef99114ca59130ccef6d0803b3 P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8c4a518929d4bb3a88bdfb1649493bd, State: Initialized, Role: FOLLOWER
I20240502 01:52:39.558794 10964 tablet_bootstrap.cc:492] T ec9d18ab22e1466c8daa6b554ccaeebb P 7b7a47245f294ce9a264fcc8f6351c0f: Bootstrap starting.
I20240502 01:52:39.559744 10963 consensus_queue.cc:260] T 8fcb99ef99114ca59130ccef6d0803b3 P a8c4a518929d4bb3a88bdfb1649493bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } }
I20240502 01:52:39.560671 10963 ts_tablet_manager.cc:1398] T 8fcb99ef99114ca59130ccef6d0803b3 P a8c4a518929d4bb3a88bdfb1649493bd: Time spent starting tablet: real 0.015s	user 0.002s	sys 0.002s
I20240502 01:52:39.564388 10964 tablet_bootstrap.cc:654] T ec9d18ab22e1466c8daa6b554ccaeebb P 7b7a47245f294ce9a264fcc8f6351c0f: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:39.567183 10967 tablet_bootstrap.cc:492] T a92d5a681b364e50896bdda0d92bb10a P fd95bcd4cdcb4cc5966050ffe2cffdff: No bootstrap required, opened a new log
I20240502 01:52:39.567365 10967 ts_tablet_manager.cc:1367] T a92d5a681b364e50896bdda0d92bb10a P fd95bcd4cdcb4cc5966050ffe2cffdff: Time spent bootstrapping tablet: real 0.033s	user 0.015s	sys 0.015s
I20240502 01:52:39.567896 10965 tablet_bootstrap.cc:492] T 8fcb99ef99114ca59130ccef6d0803b3 P a885359a82bd4515913f7b273591f851: No bootstrap required, opened a new log
I20240502 01:52:39.568131 10965 ts_tablet_manager.cc:1367] T 8fcb99ef99114ca59130ccef6d0803b3 P a885359a82bd4515913f7b273591f851: Time spent bootstrapping tablet: real 0.051s	user 0.004s	sys 0.003s
I20240502 01:52:39.568163 10967 raft_consensus.cc:348] T a92d5a681b364e50896bdda0d92bb10a P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } } peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } }
I20240502 01:52:39.568387 10967 raft_consensus.cc:374] T a92d5a681b364e50896bdda0d92bb10a P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:39.568472 10967 raft_consensus.cc:729] T a92d5a681b364e50896bdda0d92bb10a P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd95bcd4cdcb4cc5966050ffe2cffdff, State: Initialized, Role: FOLLOWER
I20240502 01:52:39.568761 10967 consensus_queue.cc:260] T a92d5a681b364e50896bdda0d92bb10a P fd95bcd4cdcb4cc5966050ffe2cffdff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } } peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } }
I20240502 01:52:39.569175 10965 raft_consensus.cc:348] T 8fcb99ef99114ca59130ccef6d0803b3 P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } }
I20240502 01:52:39.569463 10965 raft_consensus.cc:374] T 8fcb99ef99114ca59130ccef6d0803b3 P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:39.569547 10965 raft_consensus.cc:729] T 8fcb99ef99114ca59130ccef6d0803b3 P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a885359a82bd4515913f7b273591f851, State: Initialized, Role: FOLLOWER
I20240502 01:52:39.569898 10965 consensus_queue.cc:260] T 8fcb99ef99114ca59130ccef6d0803b3 P a885359a82bd4515913f7b273591f851 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } }
I20240502 01:52:39.571661 10962 tablet_bootstrap.cc:492] T ec9d18ab22e1466c8daa6b554ccaeebb P 34da849acb2045a9a2ef03933592dd08: No bootstrap required, opened a new log
I20240502 01:52:39.571879 10962 ts_tablet_manager.cc:1367] T ec9d18ab22e1466c8daa6b554ccaeebb P 34da849acb2045a9a2ef03933592dd08: Time spent bootstrapping tablet: real 0.055s	user 0.007s	sys 0.000s
I20240502 01:52:39.571921 10967 ts_tablet_manager.cc:1398] T a92d5a681b364e50896bdda0d92bb10a P fd95bcd4cdcb4cc5966050ffe2cffdff: Time spent starting tablet: real 0.004s	user 0.000s	sys 0.002s
I20240502 01:52:39.573257 10962 raft_consensus.cc:348] T ec9d18ab22e1466c8daa6b554ccaeebb P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } } peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } }
I20240502 01:52:39.573555 10962 raft_consensus.cc:374] T ec9d18ab22e1466c8daa6b554ccaeebb P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:39.573655 10962 raft_consensus.cc:729] T ec9d18ab22e1466c8daa6b554ccaeebb P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34da849acb2045a9a2ef03933592dd08, State: Initialized, Role: FOLLOWER
I20240502 01:52:39.575311 10965 ts_tablet_manager.cc:1398] T 8fcb99ef99114ca59130ccef6d0803b3 P a885359a82bd4515913f7b273591f851: Time spent starting tablet: real 0.007s	user 0.002s	sys 0.000s
I20240502 01:52:39.575659 10962 consensus_queue.cc:260] T ec9d18ab22e1466c8daa6b554ccaeebb P 34da849acb2045a9a2ef03933592dd08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } } peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } }
I20240502 01:52:39.575870 10964 tablet_bootstrap.cc:492] T ec9d18ab22e1466c8daa6b554ccaeebb P 7b7a47245f294ce9a264fcc8f6351c0f: No bootstrap required, opened a new log
I20240502 01:52:39.576062 10964 ts_tablet_manager.cc:1367] T ec9d18ab22e1466c8daa6b554ccaeebb P 7b7a47245f294ce9a264fcc8f6351c0f: Time spent bootstrapping tablet: real 0.022s	user 0.005s	sys 0.000s
I20240502 01:52:39.576884 10962 ts_tablet_manager.cc:1398] T ec9d18ab22e1466c8daa6b554ccaeebb P 34da849acb2045a9a2ef03933592dd08: Time spent starting tablet: real 0.005s	user 0.003s	sys 0.000s
I20240502 01:52:39.576840 10964 raft_consensus.cc:348] T ec9d18ab22e1466c8daa6b554ccaeebb P 7b7a47245f294ce9a264fcc8f6351c0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } } peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } }
I20240502 01:52:39.577054 10964 raft_consensus.cc:374] T ec9d18ab22e1466c8daa6b554ccaeebb P 7b7a47245f294ce9a264fcc8f6351c0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:39.577152 10964 raft_consensus.cc:729] T ec9d18ab22e1466c8daa6b554ccaeebb P 7b7a47245f294ce9a264fcc8f6351c0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b7a47245f294ce9a264fcc8f6351c0f, State: Initialized, Role: FOLLOWER
I20240502 01:52:39.577176 10962 tablet_bootstrap.cc:492] T a92d5a681b364e50896bdda0d92bb10a P 34da849acb2045a9a2ef03933592dd08: Bootstrap starting.
I20240502 01:52:39.577395 10964 consensus_queue.cc:260] T ec9d18ab22e1466c8daa6b554ccaeebb P 7b7a47245f294ce9a264fcc8f6351c0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } } peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } }
I20240502 01:52:39.577921 10964 ts_tablet_manager.cc:1398] T ec9d18ab22e1466c8daa6b554ccaeebb P 7b7a47245f294ce9a264fcc8f6351c0f: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 01:52:39.579519 10962 tablet_bootstrap.cc:654] T a92d5a681b364e50896bdda0d92bb10a P 34da849acb2045a9a2ef03933592dd08: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:39.588882 10962 tablet_bootstrap.cc:492] T a92d5a681b364e50896bdda0d92bb10a P 34da849acb2045a9a2ef03933592dd08: No bootstrap required, opened a new log
I20240502 01:52:39.589172 10962 ts_tablet_manager.cc:1367] T a92d5a681b364e50896bdda0d92bb10a P 34da849acb2045a9a2ef03933592dd08: Time spent bootstrapping tablet: real 0.012s	user 0.005s	sys 0.005s
I20240502 01:52:39.590095 10962 raft_consensus.cc:348] T a92d5a681b364e50896bdda0d92bb10a P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } } peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } }
I20240502 01:52:39.590315 10962 raft_consensus.cc:374] T a92d5a681b364e50896bdda0d92bb10a P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:39.590463 10962 raft_consensus.cc:729] T a92d5a681b364e50896bdda0d92bb10a P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34da849acb2045a9a2ef03933592dd08, State: Initialized, Role: FOLLOWER
I20240502 01:52:39.590763 10962 consensus_queue.cc:260] T a92d5a681b364e50896bdda0d92bb10a P 34da849acb2045a9a2ef03933592dd08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } } peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } }
I20240502 01:52:39.591291 10962 ts_tablet_manager.cc:1398] T a92d5a681b364e50896bdda0d92bb10a P 34da849acb2045a9a2ef03933592dd08: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240502 01:52:39.593076 10966 tablet_bootstrap.cc:492] T 8fcb99ef99114ca59130ccef6d0803b3 P ede1dd598e3541138a19a3ea5c052bc9: No bootstrap required, opened a new log
I20240502 01:52:39.593235 10966 ts_tablet_manager.cc:1367] T 8fcb99ef99114ca59130ccef6d0803b3 P ede1dd598e3541138a19a3ea5c052bc9: Time spent bootstrapping tablet: real 0.075s	user 0.014s	sys 0.031s
I20240502 01:52:39.593962 10966 raft_consensus.cc:348] T 8fcb99ef99114ca59130ccef6d0803b3 P ede1dd598e3541138a19a3ea5c052bc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } }
I20240502 01:52:39.594128 10966 raft_consensus.cc:374] T 8fcb99ef99114ca59130ccef6d0803b3 P ede1dd598e3541138a19a3ea5c052bc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:39.594228 10966 raft_consensus.cc:729] T 8fcb99ef99114ca59130ccef6d0803b3 P ede1dd598e3541138a19a3ea5c052bc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ede1dd598e3541138a19a3ea5c052bc9, State: Initialized, Role: FOLLOWER
I20240502 01:52:39.594465 10966 consensus_queue.cc:260] T 8fcb99ef99114ca59130ccef6d0803b3 P ede1dd598e3541138a19a3ea5c052bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } }
I20240502 01:52:39.594966 10966 ts_tablet_manager.cc:1398] T 8fcb99ef99114ca59130ccef6d0803b3 P ede1dd598e3541138a19a3ea5c052bc9: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 01:52:39.641212 10940 raft_consensus.cc:482] T ec9d18ab22e1466c8daa6b554ccaeebb P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:39.641409 10940 raft_consensus.cc:504] T ec9d18ab22e1466c8daa6b554ccaeebb P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } } peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } }
I20240502 01:52:39.642364 10940 leader_election.cc:288] T ec9d18ab22e1466c8daa6b554ccaeebb P fd95bcd4cdcb4cc5966050ffe2cffdff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 34da849acb2045a9a2ef03933592dd08 (127.9.176.196:44521), 7b7a47245f294ce9a264fcc8f6351c0f (127.9.176.194:43645)
I20240502 01:52:39.650614 10512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec9d18ab22e1466c8daa6b554ccaeebb" candidate_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34da849acb2045a9a2ef03933592dd08" is_pre_election: true
I20240502 01:52:39.650884 10512 raft_consensus.cc:2454] T ec9d18ab22e1466c8daa6b554ccaeebb P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd95bcd4cdcb4cc5966050ffe2cffdff in term 0.
I20240502 01:52:39.651233 10246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec9d18ab22e1466c8daa6b554ccaeebb" candidate_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" is_pre_election: true
I20240502 01:52:39.651420 10711 leader_election.cc:302] T ec9d18ab22e1466c8daa6b554ccaeebb P fd95bcd4cdcb4cc5966050ffe2cffdff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34da849acb2045a9a2ef03933592dd08, fd95bcd4cdcb4cc5966050ffe2cffdff; no voters: 
I20240502 01:52:39.651513 10246 raft_consensus.cc:2454] T ec9d18ab22e1466c8daa6b554ccaeebb P 7b7a47245f294ce9a264fcc8f6351c0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd95bcd4cdcb4cc5966050ffe2cffdff in term 0.
I20240502 01:52:39.651757 10940 raft_consensus.cc:2781] T ec9d18ab22e1466c8daa6b554ccaeebb P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:39.651883 10940 raft_consensus.cc:482] T ec9d18ab22e1466c8daa6b554ccaeebb P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:39.651980 10940 raft_consensus.cc:3037] T ec9d18ab22e1466c8daa6b554ccaeebb P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:39.653988 10940 raft_consensus.cc:504] T ec9d18ab22e1466c8daa6b554ccaeebb P fd95bcd4cdcb4cc5966050ffe2cffdff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } } peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } }
I20240502 01:52:39.654646 10940 leader_election.cc:288] T ec9d18ab22e1466c8daa6b554ccaeebb P fd95bcd4cdcb4cc5966050ffe2cffdff [CANDIDATE]: Term 1 election: Requested vote from peers 34da849acb2045a9a2ef03933592dd08 (127.9.176.196:44521), 7b7a47245f294ce9a264fcc8f6351c0f (127.9.176.194:43645)
I20240502 01:52:39.655140 10512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec9d18ab22e1466c8daa6b554ccaeebb" candidate_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34da849acb2045a9a2ef03933592dd08"
I20240502 01:52:39.655196 10246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec9d18ab22e1466c8daa6b554ccaeebb" candidate_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b7a47245f294ce9a264fcc8f6351c0f"
I20240502 01:52:39.655383 10512 raft_consensus.cc:3037] T ec9d18ab22e1466c8daa6b554ccaeebb P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:39.655439 10246 raft_consensus.cc:3037] T ec9d18ab22e1466c8daa6b554ccaeebb P 7b7a47245f294ce9a264fcc8f6351c0f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:39.657409 10512 raft_consensus.cc:2454] T ec9d18ab22e1466c8daa6b554ccaeebb P 34da849acb2045a9a2ef03933592dd08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd95bcd4cdcb4cc5966050ffe2cffdff in term 1.
I20240502 01:52:39.657409 10246 raft_consensus.cc:2454] T ec9d18ab22e1466c8daa6b554ccaeebb P 7b7a47245f294ce9a264fcc8f6351c0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd95bcd4cdcb4cc5966050ffe2cffdff in term 1.
I20240502 01:52:39.657876 10711 leader_election.cc:302] T ec9d18ab22e1466c8daa6b554ccaeebb P fd95bcd4cdcb4cc5966050ffe2cffdff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b7a47245f294ce9a264fcc8f6351c0f, fd95bcd4cdcb4cc5966050ffe2cffdff; no voters: 
I20240502 01:52:39.658238 10940 raft_consensus.cc:2781] T ec9d18ab22e1466c8daa6b554ccaeebb P fd95bcd4cdcb4cc5966050ffe2cffdff [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:39.658401 10940 raft_consensus.cc:686] T ec9d18ab22e1466c8daa6b554ccaeebb P fd95bcd4cdcb4cc5966050ffe2cffdff [term 1 LEADER]: Becoming Leader. State: Replica: fd95bcd4cdcb4cc5966050ffe2cffdff, State: Running, Role: LEADER
I20240502 01:52:39.658699 10940 consensus_queue.cc:237] T ec9d18ab22e1466c8daa6b554ccaeebb P fd95bcd4cdcb4cc5966050ffe2cffdff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } } peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } }
I20240502 01:52:39.661625  9963 catalog_manager.cc:5484] T ec9d18ab22e1466c8daa6b554ccaeebb P fd95bcd4cdcb4cc5966050ffe2cffdff reported cstate change: term changed from 0 to 1, leader changed from <none> to fd95bcd4cdcb4cc5966050ffe2cffdff (127.9.176.198). New cstate: current_term: 1 leader_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } health_report { overall_health: UNKNOWN } } }
I20240502 01:52:39.858117 10894 raft_consensus.cc:482] T 8fcb99ef99114ca59130ccef6d0803b3 P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:39.858358 10894 raft_consensus.cc:504] T 8fcb99ef99114ca59130ccef6d0803b3 P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } }
I20240502 01:52:39.858783 10940 raft_consensus.cc:482] T a92d5a681b364e50896bdda0d92bb10a P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:39.858952 10940 raft_consensus.cc:504] T a92d5a681b364e50896bdda0d92bb10a P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } } peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } }
I20240502 01:52:39.859395 10894 leader_election.cc:288] T 8fcb99ef99114ca59130ccef6d0803b3 P a8c4a518929d4bb3a88bdfb1649493bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ede1dd598e3541138a19a3ea5c052bc9 (127.9.176.195:38701), a885359a82bd4515913f7b273591f851 (127.9.176.193:38373)
I20240502 01:52:39.859685 10940 leader_election.cc:288] T a92d5a681b364e50896bdda0d92bb10a P fd95bcd4cdcb4cc5966050ffe2cffdff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b7a47245f294ce9a264fcc8f6351c0f (127.9.176.194:43645), 34da849acb2045a9a2ef03933592dd08 (127.9.176.196:44521)
I20240502 01:52:39.861079 10246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a92d5a681b364e50896bdda0d92bb10a" candidate_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" is_pre_election: true
I20240502 01:52:39.861130 10512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a92d5a681b364e50896bdda0d92bb10a" candidate_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34da849acb2045a9a2ef03933592dd08" is_pre_election: true
I20240502 01:52:39.861382 10246 raft_consensus.cc:2454] T a92d5a681b364e50896bdda0d92bb10a P 7b7a47245f294ce9a264fcc8f6351c0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd95bcd4cdcb4cc5966050ffe2cffdff in term 0.
I20240502 01:52:39.861430 10512 raft_consensus.cc:2454] T a92d5a681b364e50896bdda0d92bb10a P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd95bcd4cdcb4cc5966050ffe2cffdff in term 0.
I20240502 01:52:39.862124 10711 leader_election.cc:302] T a92d5a681b364e50896bdda0d92bb10a P fd95bcd4cdcb4cc5966050ffe2cffdff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34da849acb2045a9a2ef03933592dd08, fd95bcd4cdcb4cc5966050ffe2cffdff; no voters: 
I20240502 01:52:39.863363 10940 raft_consensus.cc:2781] T a92d5a681b364e50896bdda0d92bb10a P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:39.863459 10940 raft_consensus.cc:482] T a92d5a681b364e50896bdda0d92bb10a P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:39.863596 10940 raft_consensus.cc:3037] T a92d5a681b364e50896bdda0d92bb10a P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:39.866366 10940 raft_consensus.cc:504] T a92d5a681b364e50896bdda0d92bb10a P fd95bcd4cdcb4cc5966050ffe2cffdff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } } peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } }
I20240502 01:52:39.867202 10940 leader_election.cc:288] T a92d5a681b364e50896bdda0d92bb10a P fd95bcd4cdcb4cc5966050ffe2cffdff [CANDIDATE]: Term 1 election: Requested vote from peers 7b7a47245f294ce9a264fcc8f6351c0f (127.9.176.194:43645), 34da849acb2045a9a2ef03933592dd08 (127.9.176.196:44521)
I20240502 01:52:39.867794 10246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a92d5a681b364e50896bdda0d92bb10a" candidate_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b7a47245f294ce9a264fcc8f6351c0f"
I20240502 01:52:39.867951 10512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a92d5a681b364e50896bdda0d92bb10a" candidate_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34da849acb2045a9a2ef03933592dd08"
I20240502 01:52:39.868038 10246 raft_consensus.cc:3037] T a92d5a681b364e50896bdda0d92bb10a P 7b7a47245f294ce9a264fcc8f6351c0f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:39.868185 10512 raft_consensus.cc:3037] T a92d5a681b364e50896bdda0d92bb10a P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:39.870730 10512 raft_consensus.cc:2454] T a92d5a681b364e50896bdda0d92bb10a P 34da849acb2045a9a2ef03933592dd08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd95bcd4cdcb4cc5966050ffe2cffdff in term 1.
I20240502 01:52:39.870728 10246 raft_consensus.cc:2454] T a92d5a681b364e50896bdda0d92bb10a P 7b7a47245f294ce9a264fcc8f6351c0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd95bcd4cdcb4cc5966050ffe2cffdff in term 1.
I20240502 01:52:39.871234 10711 leader_election.cc:302] T a92d5a681b364e50896bdda0d92bb10a P fd95bcd4cdcb4cc5966050ffe2cffdff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34da849acb2045a9a2ef03933592dd08, fd95bcd4cdcb4cc5966050ffe2cffdff; no voters: 
I20240502 01:52:39.871655 10378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fcb99ef99114ca59130ccef6d0803b3" candidate_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ede1dd598e3541138a19a3ea5c052bc9" is_pre_election: true
I20240502 01:52:39.871762 10112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fcb99ef99114ca59130ccef6d0803b3" candidate_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a885359a82bd4515913f7b273591f851" is_pre_election: true
I20240502 01:52:39.871928 10378 raft_consensus.cc:2454] T 8fcb99ef99114ca59130ccef6d0803b3 P ede1dd598e3541138a19a3ea5c052bc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8c4a518929d4bb3a88bdfb1649493bd in term 0.
I20240502 01:52:39.871994 10112 raft_consensus.cc:2454] T 8fcb99ef99114ca59130ccef6d0803b3 P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8c4a518929d4bb3a88bdfb1649493bd in term 0.
I20240502 01:52:39.872337 10940 raft_consensus.cc:2781] T a92d5a681b364e50896bdda0d92bb10a P fd95bcd4cdcb4cc5966050ffe2cffdff [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:39.872426 10579 leader_election.cc:302] T 8fcb99ef99114ca59130ccef6d0803b3 P a8c4a518929d4bb3a88bdfb1649493bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a8c4a518929d4bb3a88bdfb1649493bd, ede1dd598e3541138a19a3ea5c052bc9; no voters: 
I20240502 01:52:39.872540 10940 raft_consensus.cc:686] T a92d5a681b364e50896bdda0d92bb10a P fd95bcd4cdcb4cc5966050ffe2cffdff [term 1 LEADER]: Becoming Leader. State: Replica: fd95bcd4cdcb4cc5966050ffe2cffdff, State: Running, Role: LEADER
I20240502 01:52:39.872956 10894 raft_consensus.cc:2781] T 8fcb99ef99114ca59130ccef6d0803b3 P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:39.873087 10894 raft_consensus.cc:482] T 8fcb99ef99114ca59130ccef6d0803b3 P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:39.873193 10894 raft_consensus.cc:3037] T 8fcb99ef99114ca59130ccef6d0803b3 P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:39.873139 10940 consensus_queue.cc:237] T a92d5a681b364e50896bdda0d92bb10a P fd95bcd4cdcb4cc5966050ffe2cffdff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } } peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } }
I20240502 01:52:39.875669 10894 raft_consensus.cc:504] T 8fcb99ef99114ca59130ccef6d0803b3 P a8c4a518929d4bb3a88bdfb1649493bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } }
I20240502 01:52:39.876293 10894 leader_election.cc:288] T 8fcb99ef99114ca59130ccef6d0803b3 P a8c4a518929d4bb3a88bdfb1649493bd [CANDIDATE]: Term 1 election: Requested vote from peers ede1dd598e3541138a19a3ea5c052bc9 (127.9.176.195:38701), a885359a82bd4515913f7b273591f851 (127.9.176.193:38373)
I20240502 01:52:39.876406  9963 catalog_manager.cc:5484] T a92d5a681b364e50896bdda0d92bb10a P fd95bcd4cdcb4cc5966050ffe2cffdff reported cstate change: term changed from 0 to 1, leader changed from <none> to fd95bcd4cdcb4cc5966050ffe2cffdff (127.9.176.198). New cstate: current_term: 1 leader_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } health_report { overall_health: HEALTHY } } }
I20240502 01:52:39.876796 10378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fcb99ef99114ca59130ccef6d0803b3" candidate_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ede1dd598e3541138a19a3ea5c052bc9"
I20240502 01:52:39.877018 10378 raft_consensus.cc:3037] T 8fcb99ef99114ca59130ccef6d0803b3 P ede1dd598e3541138a19a3ea5c052bc9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:39.878013 10112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fcb99ef99114ca59130ccef6d0803b3" candidate_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a885359a82bd4515913f7b273591f851"
I20240502 01:52:39.878268 10112 raft_consensus.cc:3037] T 8fcb99ef99114ca59130ccef6d0803b3 P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:39.879742 10378 raft_consensus.cc:2454] T 8fcb99ef99114ca59130ccef6d0803b3 P ede1dd598e3541138a19a3ea5c052bc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8c4a518929d4bb3a88bdfb1649493bd in term 1.
I20240502 01:52:39.880244 10579 leader_election.cc:302] T 8fcb99ef99114ca59130ccef6d0803b3 P a8c4a518929d4bb3a88bdfb1649493bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a8c4a518929d4bb3a88bdfb1649493bd, ede1dd598e3541138a19a3ea5c052bc9; no voters: 
I20240502 01:52:39.880617 10112 raft_consensus.cc:2454] T 8fcb99ef99114ca59130ccef6d0803b3 P a885359a82bd4515913f7b273591f851 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8c4a518929d4bb3a88bdfb1649493bd in term 1.
I20240502 01:52:39.880836 10902 raft_consensus.cc:2781] T 8fcb99ef99114ca59130ccef6d0803b3 P a8c4a518929d4bb3a88bdfb1649493bd [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:39.880992 10902 raft_consensus.cc:686] T 8fcb99ef99114ca59130ccef6d0803b3 P a8c4a518929d4bb3a88bdfb1649493bd [term 1 LEADER]: Becoming Leader. State: Replica: a8c4a518929d4bb3a88bdfb1649493bd, State: Running, Role: LEADER
I20240502 01:52:39.881299 10902 consensus_queue.cc:237] T 8fcb99ef99114ca59130ccef6d0803b3 P a8c4a518929d4bb3a88bdfb1649493bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } }
I20240502 01:52:39.883787  9963 catalog_manager.cc:5484] T 8fcb99ef99114ca59130ccef6d0803b3 P a8c4a518929d4bb3a88bdfb1649493bd reported cstate change: term changed from 0 to 1, leader changed from <none> to a8c4a518929d4bb3a88bdfb1649493bd (127.9.176.197). New cstate: current_term: 1 leader_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } health_report { overall_health: UNKNOWN } } }
I20240502 01:52:40.043720 10940 consensus_queue.cc:1035] T ec9d18ab22e1466c8daa6b554ccaeebb P fd95bcd4cdcb4cc5966050ffe2cffdff [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:40.059904 10881 consensus_queue.cc:1035] T ec9d18ab22e1466c8daa6b554ccaeebb P fd95bcd4cdcb4cc5966050ffe2cffdff [LEADER]: Connected to new peer: Peer: permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:40.264022 10940 consensus_queue.cc:1035] T a92d5a681b364e50896bdda0d92bb10a P fd95bcd4cdcb4cc5966050ffe2cffdff [LEADER]: Connected to new peer: Peer: permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:40.269497 10940 consensus_queue.cc:1035] T a92d5a681b364e50896bdda0d92bb10a P fd95bcd4cdcb4cc5966050ffe2cffdff [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:40.308065 10902 consensus_queue.cc:1035] T 8fcb99ef99114ca59130ccef6d0803b3 P a8c4a518929d4bb3a88bdfb1649493bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:40.315052 10902 consensus_queue.cc:1035] T 8fcb99ef99114ca59130ccef6d0803b3 P a8c4a518929d4bb3a88bdfb1649493bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:52:40.316864 10988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:40.317224 10988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:40.324995 10988 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
W20240502 01:52:40.574935 11015 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.9.176.198:33039, user_credentials={real_user=slave}} blocked reactor thread for 45807.5us

INSERT report
    rows total: 2000
    time total: 329.452 ms
  time per row: 0.164726 ms
I20240502 01:52:40.811534  9965 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56524:
name: "rebalance_test_table_2"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 01:52:40.820832 10092 tablet_service.cc:1467] Processing CreateTablet for tablet e190679362c847c58c31c6bbfc293b32 (DEFAULT_TABLE table=rebalance_test_table_2 [id=a612b8a8475c4526a8dc63891d780504]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:40.820830 10358 tablet_service.cc:1467] Processing CreateTablet for tablet e190679362c847c58c31c6bbfc293b32 (DEFAULT_TABLE table=rebalance_test_table_2 [id=a612b8a8475c4526a8dc63891d780504]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:40.821357 10092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e190679362c847c58c31c6bbfc293b32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:40.821357 10358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e190679362c847c58c31c6bbfc293b32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:40.821298 10623 tablet_service.cc:1467] Processing CreateTablet for tablet e190679362c847c58c31c6bbfc293b32 (DEFAULT_TABLE table=rebalance_test_table_2 [id=a612b8a8475c4526a8dc63891d780504]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:40.821745 10623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e190679362c847c58c31c6bbfc293b32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:40.826027 10224 tablet_service.cc:1467] Processing CreateTablet for tablet 816e798bf21641c1872dc88d2fd3e820 (DEFAULT_TABLE table=rebalance_test_table_2 [id=a612b8a8475c4526a8dc63891d780504]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:40.826509 10224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 816e798bf21641c1872dc88d2fd3e820. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:40.829226 10091 tablet_service.cc:1467] Processing CreateTablet for tablet b0522638f70b4cb28359b7233b4b6121 (DEFAULT_TABLE table=rebalance_test_table_2 [id=a612b8a8475c4526a8dc63891d780504]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:40.829759 10091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0522638f70b4cb28359b7233b4b6121. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:40.832206 10492 tablet_service.cc:1467] Processing CreateTablet for tablet 816e798bf21641c1872dc88d2fd3e820 (DEFAULT_TABLE table=rebalance_test_table_2 [id=a612b8a8475c4526a8dc63891d780504]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:40.832811 10492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 816e798bf21641c1872dc88d2fd3e820. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:40.836114 10624 tablet_service.cc:1467] Processing CreateTablet for tablet b0522638f70b4cb28359b7233b4b6121 (DEFAULT_TABLE table=rebalance_test_table_2 [id=a612b8a8475c4526a8dc63891d780504]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:40.836546 10624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0522638f70b4cb28359b7233b4b6121. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:40.836611 11053 tablet_bootstrap.cc:492] T e190679362c847c58c31c6bbfc293b32 P ede1dd598e3541138a19a3ea5c052bc9: Bootstrap starting.
I20240502 01:52:40.839520 10759 tablet_service.cc:1467] Processing CreateTablet for tablet 816e798bf21641c1872dc88d2fd3e820 (DEFAULT_TABLE table=rebalance_test_table_2 [id=a612b8a8475c4526a8dc63891d780504]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:40.840148 10759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 816e798bf21641c1872dc88d2fd3e820. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:40.845363 11052 tablet_bootstrap.cc:492] T e190679362c847c58c31c6bbfc293b32 P a8c4a518929d4bb3a88bdfb1649493bd: Bootstrap starting.
I20240502 01:52:40.847158 11054 tablet_bootstrap.cc:492] T 816e798bf21641c1872dc88d2fd3e820 P 34da849acb2045a9a2ef03933592dd08: Bootstrap starting.
I20240502 01:52:40.847551 11052 tablet_bootstrap.cc:654] T e190679362c847c58c31c6bbfc293b32 P a8c4a518929d4bb3a88bdfb1649493bd: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:40.847713 10358 tablet_service.cc:1467] Processing CreateTablet for tablet b0522638f70b4cb28359b7233b4b6121 (DEFAULT_TABLE table=rebalance_test_table_2 [id=a612b8a8475c4526a8dc63891d780504]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:40.848248 10358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0522638f70b4cb28359b7233b4b6121. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:40.848356 11055 tablet_bootstrap.cc:492] T 816e798bf21641c1872dc88d2fd3e820 P 7b7a47245f294ce9a264fcc8f6351c0f: Bootstrap starting.
I20240502 01:52:40.850728 11055 tablet_bootstrap.cc:654] T 816e798bf21641c1872dc88d2fd3e820 P 7b7a47245f294ce9a264fcc8f6351c0f: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:40.851073 11054 tablet_bootstrap.cc:654] T 816e798bf21641c1872dc88d2fd3e820 P 34da849acb2045a9a2ef03933592dd08: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:40.852507 11051 tablet_bootstrap.cc:492] T e190679362c847c58c31c6bbfc293b32 P a885359a82bd4515913f7b273591f851: Bootstrap starting.
I20240502 01:52:40.853296 11056 tablet_bootstrap.cc:492] T 816e798bf21641c1872dc88d2fd3e820 P fd95bcd4cdcb4cc5966050ffe2cffdff: Bootstrap starting.
I20240502 01:52:40.854276 11053 tablet_bootstrap.cc:654] T e190679362c847c58c31c6bbfc293b32 P ede1dd598e3541138a19a3ea5c052bc9: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:40.855518 11051 tablet_bootstrap.cc:654] T e190679362c847c58c31c6bbfc293b32 P a885359a82bd4515913f7b273591f851: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:40.857038 11056 tablet_bootstrap.cc:654] T 816e798bf21641c1872dc88d2fd3e820 P fd95bcd4cdcb4cc5966050ffe2cffdff: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:40.859757 11056 tablet_bootstrap.cc:492] T 816e798bf21641c1872dc88d2fd3e820 P fd95bcd4cdcb4cc5966050ffe2cffdff: No bootstrap required, opened a new log
I20240502 01:52:40.859944 11056 ts_tablet_manager.cc:1367] T 816e798bf21641c1872dc88d2fd3e820 P fd95bcd4cdcb4cc5966050ffe2cffdff: Time spent bootstrapping tablet: real 0.007s	user 0.004s	sys 0.000s
I20240502 01:52:40.860344 11051 tablet_bootstrap.cc:492] T e190679362c847c58c31c6bbfc293b32 P a885359a82bd4515913f7b273591f851: No bootstrap required, opened a new log
I20240502 01:52:40.860596 11051 ts_tablet_manager.cc:1367] T e190679362c847c58c31c6bbfc293b32 P a885359a82bd4515913f7b273591f851: Time spent bootstrapping tablet: real 0.008s	user 0.002s	sys 0.002s
I20240502 01:52:40.860878 11056 raft_consensus.cc:348] T 816e798bf21641c1872dc88d2fd3e820 P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } }
I20240502 01:52:40.861114 11056 raft_consensus.cc:374] T 816e798bf21641c1872dc88d2fd3e820 P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:40.861229 11056 raft_consensus.cc:729] T 816e798bf21641c1872dc88d2fd3e820 P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd95bcd4cdcb4cc5966050ffe2cffdff, State: Initialized, Role: FOLLOWER
I20240502 01:52:40.862165 11051 raft_consensus.cc:348] T e190679362c847c58c31c6bbfc293b32 P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } }
I20240502 01:52:40.862550 11051 raft_consensus.cc:374] T e190679362c847c58c31c6bbfc293b32 P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:40.862699 11051 raft_consensus.cc:729] T e190679362c847c58c31c6bbfc293b32 P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a885359a82bd4515913f7b273591f851, State: Initialized, Role: FOLLOWER
I20240502 01:52:40.862627 11056 consensus_queue.cc:260] T 816e798bf21641c1872dc88d2fd3e820 P fd95bcd4cdcb4cc5966050ffe2cffdff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } }
I20240502 01:52:40.863292 11055 tablet_bootstrap.cc:492] T 816e798bf21641c1872dc88d2fd3e820 P 7b7a47245f294ce9a264fcc8f6351c0f: No bootstrap required, opened a new log
I20240502 01:52:40.863138 11051 consensus_queue.cc:260] T e190679362c847c58c31c6bbfc293b32 P a885359a82bd4515913f7b273591f851 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } }
I20240502 01:52:40.863534 11055 ts_tablet_manager.cc:1367] T 816e798bf21641c1872dc88d2fd3e820 P 7b7a47245f294ce9a264fcc8f6351c0f: Time spent bootstrapping tablet: real 0.015s	user 0.006s	sys 0.000s
I20240502 01:52:40.863677 11052 tablet_bootstrap.cc:492] T e190679362c847c58c31c6bbfc293b32 P a8c4a518929d4bb3a88bdfb1649493bd: No bootstrap required, opened a new log
I20240502 01:52:40.863881 11051 ts_tablet_manager.cc:1398] T e190679362c847c58c31c6bbfc293b32 P a885359a82bd4515913f7b273591f851: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.001s
I20240502 01:52:40.863917 11052 ts_tablet_manager.cc:1367] T e190679362c847c58c31c6bbfc293b32 P a8c4a518929d4bb3a88bdfb1649493bd: Time spent bootstrapping tablet: real 0.019s	user 0.002s	sys 0.003s
I20240502 01:52:40.864241 11051 tablet_bootstrap.cc:492] T b0522638f70b4cb28359b7233b4b6121 P a885359a82bd4515913f7b273591f851: Bootstrap starting.
I20240502 01:52:40.864634 11055 raft_consensus.cc:348] T 816e798bf21641c1872dc88d2fd3e820 P 7b7a47245f294ce9a264fcc8f6351c0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } }
I20240502 01:52:40.864879 11055 raft_consensus.cc:374] T 816e798bf21641c1872dc88d2fd3e820 P 7b7a47245f294ce9a264fcc8f6351c0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:40.865047 11055 raft_consensus.cc:729] T 816e798bf21641c1872dc88d2fd3e820 P 7b7a47245f294ce9a264fcc8f6351c0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b7a47245f294ce9a264fcc8f6351c0f, State: Initialized, Role: FOLLOWER
I20240502 01:52:40.865473 11055 consensus_queue.cc:260] T 816e798bf21641c1872dc88d2fd3e820 P 7b7a47245f294ce9a264fcc8f6351c0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } }
I20240502 01:52:40.866386 11051 tablet_bootstrap.cc:654] T b0522638f70b4cb28359b7233b4b6121 P a885359a82bd4515913f7b273591f851: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:40.866889 11056 ts_tablet_manager.cc:1398] T 816e798bf21641c1872dc88d2fd3e820 P fd95bcd4cdcb4cc5966050ffe2cffdff: Time spent starting tablet: real 0.007s	user 0.002s	sys 0.000s
I20240502 01:52:40.870936 11053 tablet_bootstrap.cc:492] T e190679362c847c58c31c6bbfc293b32 P ede1dd598e3541138a19a3ea5c052bc9: No bootstrap required, opened a new log
I20240502 01:52:40.871309 11053 ts_tablet_manager.cc:1367] T e190679362c847c58c31c6bbfc293b32 P ede1dd598e3541138a19a3ea5c052bc9: Time spent bootstrapping tablet: real 0.035s	user 0.006s	sys 0.000s
I20240502 01:52:40.864722 11052 raft_consensus.cc:348] T e190679362c847c58c31c6bbfc293b32 P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } }
I20240502 01:52:40.872583 11052 raft_consensus.cc:374] T e190679362c847c58c31c6bbfc293b32 P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:40.872773 11052 raft_consensus.cc:729] T e190679362c847c58c31c6bbfc293b32 P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8c4a518929d4bb3a88bdfb1649493bd, State: Initialized, Role: FOLLOWER
I20240502 01:52:40.873191 11052 consensus_queue.cc:260] T e190679362c847c58c31c6bbfc293b32 P a8c4a518929d4bb3a88bdfb1649493bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } }
I20240502 01:52:40.874054 11053 raft_consensus.cc:348] T e190679362c847c58c31c6bbfc293b32 P ede1dd598e3541138a19a3ea5c052bc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } }
I20240502 01:52:40.874383 11053 raft_consensus.cc:374] T e190679362c847c58c31c6bbfc293b32 P ede1dd598e3541138a19a3ea5c052bc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:40.874537 11053 raft_consensus.cc:729] T e190679362c847c58c31c6bbfc293b32 P ede1dd598e3541138a19a3ea5c052bc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ede1dd598e3541138a19a3ea5c052bc9, State: Initialized, Role: FOLLOWER
I20240502 01:52:40.874686 11054 tablet_bootstrap.cc:492] T 816e798bf21641c1872dc88d2fd3e820 P 34da849acb2045a9a2ef03933592dd08: No bootstrap required, opened a new log
I20240502 01:52:40.874948 11054 ts_tablet_manager.cc:1367] T 816e798bf21641c1872dc88d2fd3e820 P 34da849acb2045a9a2ef03933592dd08: Time spent bootstrapping tablet: real 0.028s	user 0.004s	sys 0.008s
I20240502 01:52:40.874784 11051 tablet_bootstrap.cc:492] T b0522638f70b4cb28359b7233b4b6121 P a885359a82bd4515913f7b273591f851: No bootstrap required, opened a new log
I20240502 01:52:40.874949 11053 consensus_queue.cc:260] T e190679362c847c58c31c6bbfc293b32 P ede1dd598e3541138a19a3ea5c052bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } }
I20240502 01:52:40.875358 11051 ts_tablet_manager.cc:1367] T b0522638f70b4cb28359b7233b4b6121 P a885359a82bd4515913f7b273591f851: Time spent bootstrapping tablet: real 0.011s	user 0.010s	sys 0.000s
I20240502 01:52:40.876224 11054 raft_consensus.cc:348] T 816e798bf21641c1872dc88d2fd3e820 P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } }
I20240502 01:52:40.876574 11054 raft_consensus.cc:374] T 816e798bf21641c1872dc88d2fd3e820 P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:40.876714 11054 raft_consensus.cc:729] T 816e798bf21641c1872dc88d2fd3e820 P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34da849acb2045a9a2ef03933592dd08, State: Initialized, Role: FOLLOWER
I20240502 01:52:40.876499 11051 raft_consensus.cc:348] T b0522638f70b4cb28359b7233b4b6121 P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } }
I20240502 01:52:40.877089 11051 raft_consensus.cc:374] T b0522638f70b4cb28359b7233b4b6121 P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:40.877230 11051 raft_consensus.cc:729] T b0522638f70b4cb28359b7233b4b6121 P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a885359a82bd4515913f7b273591f851, State: Initialized, Role: FOLLOWER
I20240502 01:52:40.877617 11051 consensus_queue.cc:260] T b0522638f70b4cb28359b7233b4b6121 P a885359a82bd4515913f7b273591f851 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } }
I20240502 01:52:40.877817 11053 ts_tablet_manager.cc:1398] T e190679362c847c58c31c6bbfc293b32 P ede1dd598e3541138a19a3ea5c052bc9: Time spent starting tablet: real 0.006s	user 0.000s	sys 0.004s
I20240502 01:52:40.878129 11053 tablet_bootstrap.cc:492] T b0522638f70b4cb28359b7233b4b6121 P ede1dd598e3541138a19a3ea5c052bc9: Bootstrap starting.
I20240502 01:52:40.879235 11054 consensus_queue.cc:260] T 816e798bf21641c1872dc88d2fd3e820 P 34da849acb2045a9a2ef03933592dd08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } }
I20240502 01:52:40.880059 11053 tablet_bootstrap.cc:654] T b0522638f70b4cb28359b7233b4b6121 P ede1dd598e3541138a19a3ea5c052bc9: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:40.882045 11051 ts_tablet_manager.cc:1398] T b0522638f70b4cb28359b7233b4b6121 P a885359a82bd4515913f7b273591f851: Time spent starting tablet: real 0.007s	user 0.000s	sys 0.002s
I20240502 01:52:40.882838 11055 ts_tablet_manager.cc:1398] T 816e798bf21641c1872dc88d2fd3e820 P 7b7a47245f294ce9a264fcc8f6351c0f: Time spent starting tablet: real 0.019s	user 0.016s	sys 0.000s
I20240502 01:52:40.884008 11052 ts_tablet_manager.cc:1398] T e190679362c847c58c31c6bbfc293b32 P a8c4a518929d4bb3a88bdfb1649493bd: Time spent starting tablet: real 0.020s	user 0.001s	sys 0.001s
I20240502 01:52:40.885327 11052 tablet_bootstrap.cc:492] T b0522638f70b4cb28359b7233b4b6121 P a8c4a518929d4bb3a88bdfb1649493bd: Bootstrap starting.
I20240502 01:52:40.885994 11053 tablet_bootstrap.cc:492] T b0522638f70b4cb28359b7233b4b6121 P ede1dd598e3541138a19a3ea5c052bc9: No bootstrap required, opened a new log
I20240502 01:52:40.886171 11053 ts_tablet_manager.cc:1367] T b0522638f70b4cb28359b7233b4b6121 P ede1dd598e3541138a19a3ea5c052bc9: Time spent bootstrapping tablet: real 0.008s	user 0.002s	sys 0.002s
I20240502 01:52:40.887109 11053 raft_consensus.cc:348] T b0522638f70b4cb28359b7233b4b6121 P ede1dd598e3541138a19a3ea5c052bc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } }
I20240502 01:52:40.887403 11053 raft_consensus.cc:374] T b0522638f70b4cb28359b7233b4b6121 P ede1dd598e3541138a19a3ea5c052bc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:40.887480 11053 raft_consensus.cc:729] T b0522638f70b4cb28359b7233b4b6121 P ede1dd598e3541138a19a3ea5c052bc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ede1dd598e3541138a19a3ea5c052bc9, State: Initialized, Role: FOLLOWER
I20240502 01:52:40.887795 11053 consensus_queue.cc:260] T b0522638f70b4cb28359b7233b4b6121 P ede1dd598e3541138a19a3ea5c052bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } }
I20240502 01:52:40.888695 11053 ts_tablet_manager.cc:1398] T b0522638f70b4cb28359b7233b4b6121 P ede1dd598e3541138a19a3ea5c052bc9: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.001s
I20240502 01:52:40.889441 11052 tablet_bootstrap.cc:654] T b0522638f70b4cb28359b7233b4b6121 P a8c4a518929d4bb3a88bdfb1649493bd: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:40.892796 11052 tablet_bootstrap.cc:492] T b0522638f70b4cb28359b7233b4b6121 P a8c4a518929d4bb3a88bdfb1649493bd: No bootstrap required, opened a new log
I20240502 01:52:40.892859 11054 ts_tablet_manager.cc:1398] T 816e798bf21641c1872dc88d2fd3e820 P 34da849acb2045a9a2ef03933592dd08: Time spent starting tablet: real 0.018s	user 0.008s	sys 0.004s
I20240502 01:52:40.892995 11052 ts_tablet_manager.cc:1367] T b0522638f70b4cb28359b7233b4b6121 P a8c4a518929d4bb3a88bdfb1649493bd: Time spent bootstrapping tablet: real 0.008s	user 0.002s	sys 0.002s
I20240502 01:52:40.893878 11052 raft_consensus.cc:348] T b0522638f70b4cb28359b7233b4b6121 P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } }
I20240502 01:52:40.894122 11052 raft_consensus.cc:374] T b0522638f70b4cb28359b7233b4b6121 P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:40.894219 11052 raft_consensus.cc:729] T b0522638f70b4cb28359b7233b4b6121 P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8c4a518929d4bb3a88bdfb1649493bd, State: Initialized, Role: FOLLOWER
I20240502 01:52:40.894480 11052 consensus_queue.cc:260] T b0522638f70b4cb28359b7233b4b6121 P a8c4a518929d4bb3a88bdfb1649493bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } }
I20240502 01:52:40.894968 11052 ts_tablet_manager.cc:1398] T b0522638f70b4cb28359b7233b4b6121 P a8c4a518929d4bb3a88bdfb1649493bd: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 01:52:40.928906 10942 raft_consensus.cc:482] T b0522638f70b4cb28359b7233b4b6121 P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:40.929100 10942 raft_consensus.cc:504] T b0522638f70b4cb28359b7233b4b6121 P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } }
I20240502 01:52:40.929848 10942 leader_election.cc:288] T b0522638f70b4cb28359b7233b4b6121 P a885359a82bd4515913f7b273591f851 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ede1dd598e3541138a19a3ea5c052bc9 (127.9.176.195:38701), a8c4a518929d4bb3a88bdfb1649493bd (127.9.176.197:44061)
I20240502 01:52:40.930282 10378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0522638f70b4cb28359b7233b4b6121" candidate_uuid: "a885359a82bd4515913f7b273591f851" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ede1dd598e3541138a19a3ea5c052bc9" is_pre_election: true
I20240502 01:52:40.930544 10378 raft_consensus.cc:2454] T b0522638f70b4cb28359b7233b4b6121 P ede1dd598e3541138a19a3ea5c052bc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a885359a82bd4515913f7b273591f851 in term 0.
I20240502 01:52:40.930465 10643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0522638f70b4cb28359b7233b4b6121" candidate_uuid: "a885359a82bd4515913f7b273591f851" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" is_pre_election: true
I20240502 01:52:40.930698 10643 raft_consensus.cc:2454] T b0522638f70b4cb28359b7233b4b6121 P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a885359a82bd4515913f7b273591f851 in term 0.
I20240502 01:52:40.931041 10045 leader_election.cc:302] T b0522638f70b4cb28359b7233b4b6121 P a885359a82bd4515913f7b273591f851 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a885359a82bd4515913f7b273591f851, ede1dd598e3541138a19a3ea5c052bc9; no voters: 
I20240502 01:52:40.931397 10942 raft_consensus.cc:2781] T b0522638f70b4cb28359b7233b4b6121 P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:40.931530 10942 raft_consensus.cc:482] T b0522638f70b4cb28359b7233b4b6121 P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:40.931653 10942 raft_consensus.cc:3037] T b0522638f70b4cb28359b7233b4b6121 P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:40.933710 10942 raft_consensus.cc:504] T b0522638f70b4cb28359b7233b4b6121 P a885359a82bd4515913f7b273591f851 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } }
I20240502 01:52:40.934449 10942 leader_election.cc:288] T b0522638f70b4cb28359b7233b4b6121 P a885359a82bd4515913f7b273591f851 [CANDIDATE]: Term 1 election: Requested vote from peers ede1dd598e3541138a19a3ea5c052bc9 (127.9.176.195:38701), a8c4a518929d4bb3a88bdfb1649493bd (127.9.176.197:44061)
I20240502 01:52:40.935073 10378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0522638f70b4cb28359b7233b4b6121" candidate_uuid: "a885359a82bd4515913f7b273591f851" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ede1dd598e3541138a19a3ea5c052bc9"
I20240502 01:52:40.935101 10643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0522638f70b4cb28359b7233b4b6121" candidate_uuid: "a885359a82bd4515913f7b273591f851" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8c4a518929d4bb3a88bdfb1649493bd"
I20240502 01:52:40.935281 10643 raft_consensus.cc:3037] T b0522638f70b4cb28359b7233b4b6121 P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:40.935310 10378 raft_consensus.cc:3037] T b0522638f70b4cb28359b7233b4b6121 P ede1dd598e3541138a19a3ea5c052bc9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:40.937389 10378 raft_consensus.cc:2454] T b0522638f70b4cb28359b7233b4b6121 P ede1dd598e3541138a19a3ea5c052bc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a885359a82bd4515913f7b273591f851 in term 1.
I20240502 01:52:40.937454 10643 raft_consensus.cc:2454] T b0522638f70b4cb28359b7233b4b6121 P a8c4a518929d4bb3a88bdfb1649493bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a885359a82bd4515913f7b273591f851 in term 1.
I20240502 01:52:40.937908 10045 leader_election.cc:302] T b0522638f70b4cb28359b7233b4b6121 P a885359a82bd4515913f7b273591f851 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a885359a82bd4515913f7b273591f851, ede1dd598e3541138a19a3ea5c052bc9; no voters: 
I20240502 01:52:40.938261 10942 raft_consensus.cc:2781] T b0522638f70b4cb28359b7233b4b6121 P a885359a82bd4515913f7b273591f851 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:40.938696 10942 raft_consensus.cc:686] T b0522638f70b4cb28359b7233b4b6121 P a885359a82bd4515913f7b273591f851 [term 1 LEADER]: Becoming Leader. State: Replica: a885359a82bd4515913f7b273591f851, State: Running, Role: LEADER
I20240502 01:52:40.939060 10942 consensus_queue.cc:237] T b0522638f70b4cb28359b7233b4b6121 P a885359a82bd4515913f7b273591f851 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } }
I20240502 01:52:40.942163  9960 catalog_manager.cc:5484] T b0522638f70b4cb28359b7233b4b6121 P a885359a82bd4515913f7b273591f851 reported cstate change: term changed from 0 to 1, leader changed from <none> to a885359a82bd4515913f7b273591f851 (127.9.176.193). New cstate: current_term: 1 leader_uuid: "a885359a82bd4515913f7b273591f851" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } health_report { overall_health: HEALTHY } } }
I20240502 01:52:40.950722 10942 raft_consensus.cc:482] T e190679362c847c58c31c6bbfc293b32 P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:40.950963 10942 raft_consensus.cc:504] T e190679362c847c58c31c6bbfc293b32 P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } }
I20240502 01:52:40.951622 10942 leader_election.cc:288] T e190679362c847c58c31c6bbfc293b32 P a885359a82bd4515913f7b273591f851 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ede1dd598e3541138a19a3ea5c052bc9 (127.9.176.195:38701), a8c4a518929d4bb3a88bdfb1649493bd (127.9.176.197:44061)
I20240502 01:52:40.952194 10643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e190679362c847c58c31c6bbfc293b32" candidate_uuid: "a885359a82bd4515913f7b273591f851" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" is_pre_election: true
I20240502 01:52:40.952219 10378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e190679362c847c58c31c6bbfc293b32" candidate_uuid: "a885359a82bd4515913f7b273591f851" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ede1dd598e3541138a19a3ea5c052bc9" is_pre_election: true
I20240502 01:52:40.952446 10643 raft_consensus.cc:2454] T e190679362c847c58c31c6bbfc293b32 P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a885359a82bd4515913f7b273591f851 in term 0.
I20240502 01:52:40.952445 10378 raft_consensus.cc:2454] T e190679362c847c58c31c6bbfc293b32 P ede1dd598e3541138a19a3ea5c052bc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a885359a82bd4515913f7b273591f851 in term 0.
I20240502 01:52:40.952890 10045 leader_election.cc:302] T e190679362c847c58c31c6bbfc293b32 P a885359a82bd4515913f7b273591f851 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a885359a82bd4515913f7b273591f851, ede1dd598e3541138a19a3ea5c052bc9; no voters: 
I20240502 01:52:40.953207 10942 raft_consensus.cc:2781] T e190679362c847c58c31c6bbfc293b32 P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:40.953292 10942 raft_consensus.cc:482] T e190679362c847c58c31c6bbfc293b32 P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:40.953369 10942 raft_consensus.cc:3037] T e190679362c847c58c31c6bbfc293b32 P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:40.955273 10942 raft_consensus.cc:504] T e190679362c847c58c31c6bbfc293b32 P a885359a82bd4515913f7b273591f851 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } }
I20240502 01:52:40.955948 10942 leader_election.cc:288] T e190679362c847c58c31c6bbfc293b32 P a885359a82bd4515913f7b273591f851 [CANDIDATE]: Term 1 election: Requested vote from peers ede1dd598e3541138a19a3ea5c052bc9 (127.9.176.195:38701), a8c4a518929d4bb3a88bdfb1649493bd (127.9.176.197:44061)
I20240502 01:52:40.956486 10643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e190679362c847c58c31c6bbfc293b32" candidate_uuid: "a885359a82bd4515913f7b273591f851" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8c4a518929d4bb3a88bdfb1649493bd"
I20240502 01:52:40.956508 10378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e190679362c847c58c31c6bbfc293b32" candidate_uuid: "a885359a82bd4515913f7b273591f851" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ede1dd598e3541138a19a3ea5c052bc9"
I20240502 01:52:40.956715 10378 raft_consensus.cc:3037] T e190679362c847c58c31c6bbfc293b32 P ede1dd598e3541138a19a3ea5c052bc9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:40.956727 10643 raft_consensus.cc:3037] T e190679362c847c58c31c6bbfc293b32 P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:40.958765 10643 raft_consensus.cc:2454] T e190679362c847c58c31c6bbfc293b32 P a8c4a518929d4bb3a88bdfb1649493bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a885359a82bd4515913f7b273591f851 in term 1.
I20240502 01:52:40.958781 10378 raft_consensus.cc:2454] T e190679362c847c58c31c6bbfc293b32 P ede1dd598e3541138a19a3ea5c052bc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a885359a82bd4515913f7b273591f851 in term 1.
I20240502 01:52:40.959316 10046 leader_election.cc:302] T e190679362c847c58c31c6bbfc293b32 P a885359a82bd4515913f7b273591f851 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a885359a82bd4515913f7b273591f851, a8c4a518929d4bb3a88bdfb1649493bd; no voters: 
I20240502 01:52:40.959664 10942 raft_consensus.cc:2781] T e190679362c847c58c31c6bbfc293b32 P a885359a82bd4515913f7b273591f851 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:40.959848 10942 raft_consensus.cc:686] T e190679362c847c58c31c6bbfc293b32 P a885359a82bd4515913f7b273591f851 [term 1 LEADER]: Becoming Leader. State: Replica: a885359a82bd4515913f7b273591f851, State: Running, Role: LEADER
I20240502 01:52:40.960157 10942 consensus_queue.cc:237] T e190679362c847c58c31c6bbfc293b32 P a885359a82bd4515913f7b273591f851 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } }
I20240502 01:52:40.962966  9960 catalog_manager.cc:5484] T e190679362c847c58c31c6bbfc293b32 P a885359a82bd4515913f7b273591f851 reported cstate change: term changed from 0 to 1, leader changed from <none> to a885359a82bd4515913f7b273591f851 (127.9.176.193). New cstate: current_term: 1 leader_uuid: "a885359a82bd4515913f7b273591f851" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } health_report { overall_health: UNKNOWN } } }
I20240502 01:52:41.194659 10881 raft_consensus.cc:482] T 816e798bf21641c1872dc88d2fd3e820 P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:41.194890 10881 raft_consensus.cc:504] T 816e798bf21641c1872dc88d2fd3e820 P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } }
I20240502 01:52:41.195494 10881 leader_election.cc:288] T 816e798bf21641c1872dc88d2fd3e820 P fd95bcd4cdcb4cc5966050ffe2cffdff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 34da849acb2045a9a2ef03933592dd08 (127.9.176.196:44521), 7b7a47245f294ce9a264fcc8f6351c0f (127.9.176.194:43645)
I20240502 01:52:41.196206 10245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "816e798bf21641c1872dc88d2fd3e820" candidate_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" is_pre_election: true
I20240502 01:52:41.196214 10511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "816e798bf21641c1872dc88d2fd3e820" candidate_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34da849acb2045a9a2ef03933592dd08" is_pre_election: true
I20240502 01:52:41.196465 10245 raft_consensus.cc:2454] T 816e798bf21641c1872dc88d2fd3e820 P 7b7a47245f294ce9a264fcc8f6351c0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd95bcd4cdcb4cc5966050ffe2cffdff in term 0.
I20240502 01:52:41.196465 10511 raft_consensus.cc:2454] T 816e798bf21641c1872dc88d2fd3e820 P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd95bcd4cdcb4cc5966050ffe2cffdff in term 0.
I20240502 01:52:41.196914 10711 leader_election.cc:302] T 816e798bf21641c1872dc88d2fd3e820 P fd95bcd4cdcb4cc5966050ffe2cffdff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34da849acb2045a9a2ef03933592dd08, fd95bcd4cdcb4cc5966050ffe2cffdff; no voters: 
I20240502 01:52:41.197186 10881 raft_consensus.cc:2781] T 816e798bf21641c1872dc88d2fd3e820 P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:41.197314 10881 raft_consensus.cc:482] T 816e798bf21641c1872dc88d2fd3e820 P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:41.197420 10881 raft_consensus.cc:3037] T 816e798bf21641c1872dc88d2fd3e820 P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:41.199134 10881 raft_consensus.cc:504] T 816e798bf21641c1872dc88d2fd3e820 P fd95bcd4cdcb4cc5966050ffe2cffdff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } }
I20240502 01:52:41.199738 10881 leader_election.cc:288] T 816e798bf21641c1872dc88d2fd3e820 P fd95bcd4cdcb4cc5966050ffe2cffdff [CANDIDATE]: Term 1 election: Requested vote from peers 34da849acb2045a9a2ef03933592dd08 (127.9.176.196:44521), 7b7a47245f294ce9a264fcc8f6351c0f (127.9.176.194:43645)
I20240502 01:52:41.200125 10511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "816e798bf21641c1872dc88d2fd3e820" candidate_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34da849acb2045a9a2ef03933592dd08"
I20240502 01:52:41.200187 10245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "816e798bf21641c1872dc88d2fd3e820" candidate_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b7a47245f294ce9a264fcc8f6351c0f"
I20240502 01:52:41.200345 10511 raft_consensus.cc:3037] T 816e798bf21641c1872dc88d2fd3e820 P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:41.200384 10245 raft_consensus.cc:3037] T 816e798bf21641c1872dc88d2fd3e820 P 7b7a47245f294ce9a264fcc8f6351c0f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:41.203047 10245 raft_consensus.cc:2454] T 816e798bf21641c1872dc88d2fd3e820 P 7b7a47245f294ce9a264fcc8f6351c0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd95bcd4cdcb4cc5966050ffe2cffdff in term 1.
I20240502 01:52:41.203284 10511 raft_consensus.cc:2454] T 816e798bf21641c1872dc88d2fd3e820 P 34da849acb2045a9a2ef03933592dd08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd95bcd4cdcb4cc5966050ffe2cffdff in term 1.
I20240502 01:52:41.203470 10711 leader_election.cc:302] T 816e798bf21641c1872dc88d2fd3e820 P fd95bcd4cdcb4cc5966050ffe2cffdff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b7a47245f294ce9a264fcc8f6351c0f, fd95bcd4cdcb4cc5966050ffe2cffdff; no voters: 
I20240502 01:52:41.203780 10881 raft_consensus.cc:2781] T 816e798bf21641c1872dc88d2fd3e820 P fd95bcd4cdcb4cc5966050ffe2cffdff [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:41.204098 10881 raft_consensus.cc:686] T 816e798bf21641c1872dc88d2fd3e820 P fd95bcd4cdcb4cc5966050ffe2cffdff [term 1 LEADER]: Becoming Leader. State: Replica: fd95bcd4cdcb4cc5966050ffe2cffdff, State: Running, Role: LEADER
I20240502 01:52:41.204396 10881 consensus_queue.cc:237] T 816e798bf21641c1872dc88d2fd3e820 P fd95bcd4cdcb4cc5966050ffe2cffdff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } }
I20240502 01:52:41.206990  9960 catalog_manager.cc:5484] T 816e798bf21641c1872dc88d2fd3e820 P fd95bcd4cdcb4cc5966050ffe2cffdff reported cstate change: term changed from 0 to 1, leader changed from <none> to fd95bcd4cdcb4cc5966050ffe2cffdff (127.9.176.198). New cstate: current_term: 1 leader_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } health_report { overall_health: UNKNOWN } } }
I20240502 01:52:41.416549 10942 consensus_queue.cc:1035] T b0522638f70b4cb28359b7233b4b6121 P a885359a82bd4515913f7b273591f851 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:41.429766 11070 consensus_queue.cc:1035] T b0522638f70b4cb28359b7233b4b6121 P a885359a82bd4515913f7b273591f851 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:41.438611 11070 consensus_queue.cc:1035] T e190679362c847c58c31c6bbfc293b32 P a885359a82bd4515913f7b273591f851 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:41.448160 11070 consensus_queue.cc:1035] T e190679362c847c58c31c6bbfc293b32 P a885359a82bd4515913f7b273591f851 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:52:41.561719 11075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:41.562076 11075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:41.570027 11075 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240502 01:52:41.712018 10881 consensus_queue.cc:1035] T 816e798bf21641c1872dc88d2fd3e820 P fd95bcd4cdcb4cc5966050ffe2cffdff [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:41.770511 10881 consensus_queue.cc:1035] T 816e798bf21641c1872dc88d2fd3e820 P fd95bcd4cdcb4cc5966050ffe2cffdff [LEADER]: Connected to new peer: Peer: permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 344.595 ms
  time per row: 0.172297 ms
I20240502 01:52:42.094236  9960 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56524:
name: "rebalance_test_table_3"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 01:52:42.166851 10759 tablet_service.cc:1467] Processing CreateTablet for tablet c96831f69a9b452c811152c9b0de7588 (DEFAULT_TABLE table=rebalance_test_table_3 [id=b471b53fe9604c478e934c511daecd6c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:42.166952 10224 tablet_service.cc:1467] Processing CreateTablet for tablet c96831f69a9b452c811152c9b0de7588 (DEFAULT_TABLE table=rebalance_test_table_3 [id=b471b53fe9604c478e934c511daecd6c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:42.167474 10759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c96831f69a9b452c811152c9b0de7588. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:42.167486 10224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c96831f69a9b452c811152c9b0de7588. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:42.168272 10225 tablet_service.cc:1467] Processing CreateTablet for tablet c264070710e8405ab9c2b3fe2cea46ae (DEFAULT_TABLE table=rebalance_test_table_3 [id=b471b53fe9604c478e934c511daecd6c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:42.168715 10225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c264070710e8405ab9c2b3fe2cea46ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:42.168952 10492 tablet_service.cc:1467] Processing CreateTablet for tablet c96831f69a9b452c811152c9b0de7588 (DEFAULT_TABLE table=rebalance_test_table_3 [id=b471b53fe9604c478e934c511daecd6c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:42.169401 10492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c96831f69a9b452c811152c9b0de7588. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:42.171393 10092 tablet_service.cc:1467] Processing CreateTablet for tablet 930071b72771448d98cbe659ee23c63a (DEFAULT_TABLE table=rebalance_test_table_3 [id=b471b53fe9604c478e934c511daecd6c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:42.171913 10092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 930071b72771448d98cbe659ee23c63a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:42.172475 10623 tablet_service.cc:1467] Processing CreateTablet for tablet 930071b72771448d98cbe659ee23c63a (DEFAULT_TABLE table=rebalance_test_table_3 [id=b471b53fe9604c478e934c511daecd6c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:42.172947 10623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 930071b72771448d98cbe659ee23c63a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:42.176035 10358 tablet_service.cc:1467] Processing CreateTablet for tablet 930071b72771448d98cbe659ee23c63a (DEFAULT_TABLE table=rebalance_test_table_3 [id=b471b53fe9604c478e934c511daecd6c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:42.176561 10358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 930071b72771448d98cbe659ee23c63a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:42.178748 10491 tablet_service.cc:1467] Processing CreateTablet for tablet c264070710e8405ab9c2b3fe2cea46ae (DEFAULT_TABLE table=rebalance_test_table_3 [id=b471b53fe9604c478e934c511daecd6c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:42.179244 10491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c264070710e8405ab9c2b3fe2cea46ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:42.180276 10758 tablet_service.cc:1467] Processing CreateTablet for tablet c264070710e8405ab9c2b3fe2cea46ae (DEFAULT_TABLE table=rebalance_test_table_3 [id=b471b53fe9604c478e934c511daecd6c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:42.180768 10758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c264070710e8405ab9c2b3fe2cea46ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:42.187809 11155 tablet_bootstrap.cc:492] T 930071b72771448d98cbe659ee23c63a P a8c4a518929d4bb3a88bdfb1649493bd: Bootstrap starting.
I20240502 01:52:42.189397 11156 tablet_bootstrap.cc:492] T c96831f69a9b452c811152c9b0de7588 P fd95bcd4cdcb4cc5966050ffe2cffdff: Bootstrap starting.
I20240502 01:52:42.191820 11155 tablet_bootstrap.cc:654] T 930071b72771448d98cbe659ee23c63a P a8c4a518929d4bb3a88bdfb1649493bd: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:42.193071 11156 tablet_bootstrap.cc:654] T c96831f69a9b452c811152c9b0de7588 P fd95bcd4cdcb4cc5966050ffe2cffdff: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:42.194813 11153 tablet_bootstrap.cc:492] T c96831f69a9b452c811152c9b0de7588 P 7b7a47245f294ce9a264fcc8f6351c0f: Bootstrap starting.
I20240502 01:52:42.197333 11153 tablet_bootstrap.cc:654] T c96831f69a9b452c811152c9b0de7588 P 7b7a47245f294ce9a264fcc8f6351c0f: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:42.198483 11157 tablet_bootstrap.cc:492] T 930071b72771448d98cbe659ee23c63a P ede1dd598e3541138a19a3ea5c052bc9: Bootstrap starting.
I20240502 01:52:42.201186 11157 tablet_bootstrap.cc:654] T 930071b72771448d98cbe659ee23c63a P ede1dd598e3541138a19a3ea5c052bc9: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:42.202813 11155 tablet_bootstrap.cc:492] T 930071b72771448d98cbe659ee23c63a P a8c4a518929d4bb3a88bdfb1649493bd: No bootstrap required, opened a new log
I20240502 01:52:42.203073 11155 ts_tablet_manager.cc:1367] T 930071b72771448d98cbe659ee23c63a P a8c4a518929d4bb3a88bdfb1649493bd: Time spent bootstrapping tablet: real 0.016s	user 0.002s	sys 0.007s
I20240502 01:52:42.203495 11158 tablet_bootstrap.cc:492] T c96831f69a9b452c811152c9b0de7588 P 34da849acb2045a9a2ef03933592dd08: Bootstrap starting.
I20240502 01:52:42.205492 11153 tablet_bootstrap.cc:492] T c96831f69a9b452c811152c9b0de7588 P 7b7a47245f294ce9a264fcc8f6351c0f: No bootstrap required, opened a new log
I20240502 01:52:42.205593 11155 raft_consensus.cc:348] T 930071b72771448d98cbe659ee23c63a P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } }
I20240502 01:52:42.206045 11155 raft_consensus.cc:374] T 930071b72771448d98cbe659ee23c63a P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:42.206231 11155 raft_consensus.cc:729] T 930071b72771448d98cbe659ee23c63a P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8c4a518929d4bb3a88bdfb1649493bd, State: Initialized, Role: FOLLOWER
I20240502 01:52:42.206367 11158 tablet_bootstrap.cc:654] T c96831f69a9b452c811152c9b0de7588 P 34da849acb2045a9a2ef03933592dd08: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:42.206550 11153 ts_tablet_manager.cc:1367] T c96831f69a9b452c811152c9b0de7588 P 7b7a47245f294ce9a264fcc8f6351c0f: Time spent bootstrapping tablet: real 0.012s	user 0.006s	sys 0.000s
I20240502 01:52:42.206754 11155 consensus_queue.cc:260] T 930071b72771448d98cbe659ee23c63a P a8c4a518929d4bb3a88bdfb1649493bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } }
I20240502 01:52:42.207984 11154 tablet_bootstrap.cc:492] T 930071b72771448d98cbe659ee23c63a P a885359a82bd4515913f7b273591f851: Bootstrap starting.
I20240502 01:52:42.208012 11153 raft_consensus.cc:348] T c96831f69a9b452c811152c9b0de7588 P 7b7a47245f294ce9a264fcc8f6351c0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } }
I20240502 01:52:42.209288 11153 raft_consensus.cc:374] T c96831f69a9b452c811152c9b0de7588 P 7b7a47245f294ce9a264fcc8f6351c0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:42.209619 11153 raft_consensus.cc:729] T c96831f69a9b452c811152c9b0de7588 P 7b7a47245f294ce9a264fcc8f6351c0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b7a47245f294ce9a264fcc8f6351c0f, State: Initialized, Role: FOLLOWER
I20240502 01:52:42.210152 11153 consensus_queue.cc:260] T c96831f69a9b452c811152c9b0de7588 P 7b7a47245f294ce9a264fcc8f6351c0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } }
I20240502 01:52:42.210369 11154 tablet_bootstrap.cc:654] T 930071b72771448d98cbe659ee23c63a P a885359a82bd4515913f7b273591f851: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:42.211422 11153 ts_tablet_manager.cc:1398] T c96831f69a9b452c811152c9b0de7588 P 7b7a47245f294ce9a264fcc8f6351c0f: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.000s
I20240502 01:52:42.211879 11153 tablet_bootstrap.cc:492] T c264070710e8405ab9c2b3fe2cea46ae P 7b7a47245f294ce9a264fcc8f6351c0f: Bootstrap starting.
I20240502 01:52:42.213300 11156 tablet_bootstrap.cc:492] T c96831f69a9b452c811152c9b0de7588 P fd95bcd4cdcb4cc5966050ffe2cffdff: No bootstrap required, opened a new log
I20240502 01:52:42.213552 11156 ts_tablet_manager.cc:1367] T c96831f69a9b452c811152c9b0de7588 P fd95bcd4cdcb4cc5966050ffe2cffdff: Time spent bootstrapping tablet: real 0.024s	user 0.003s	sys 0.008s
I20240502 01:52:42.213699 11155 ts_tablet_manager.cc:1398] T 930071b72771448d98cbe659ee23c63a P a8c4a518929d4bb3a88bdfb1649493bd: Time spent starting tablet: real 0.010s	user 0.003s	sys 0.000s
I20240502 01:52:42.214514 11156 raft_consensus.cc:348] T c96831f69a9b452c811152c9b0de7588 P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } }
I20240502 01:52:42.214855 11156 raft_consensus.cc:374] T c96831f69a9b452c811152c9b0de7588 P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:42.215004 11156 raft_consensus.cc:729] T c96831f69a9b452c811152c9b0de7588 P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd95bcd4cdcb4cc5966050ffe2cffdff, State: Initialized, Role: FOLLOWER
I20240502 01:52:42.215361 11156 consensus_queue.cc:260] T c96831f69a9b452c811152c9b0de7588 P fd95bcd4cdcb4cc5966050ffe2cffdff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } }
I20240502 01:52:42.216238 11156 ts_tablet_manager.cc:1398] T c96831f69a9b452c811152c9b0de7588 P fd95bcd4cdcb4cc5966050ffe2cffdff: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 01:52:42.216593 11156 tablet_bootstrap.cc:492] T c264070710e8405ab9c2b3fe2cea46ae P fd95bcd4cdcb4cc5966050ffe2cffdff: Bootstrap starting.
I20240502 01:52:42.218364 11157 tablet_bootstrap.cc:492] T 930071b72771448d98cbe659ee23c63a P ede1dd598e3541138a19a3ea5c052bc9: No bootstrap required, opened a new log
I20240502 01:52:42.218533 11157 ts_tablet_manager.cc:1367] T 930071b72771448d98cbe659ee23c63a P ede1dd598e3541138a19a3ea5c052bc9: Time spent bootstrapping tablet: real 0.020s	user 0.004s	sys 0.012s
I20240502 01:52:42.218626 11158 tablet_bootstrap.cc:492] T c96831f69a9b452c811152c9b0de7588 P 34da849acb2045a9a2ef03933592dd08: No bootstrap required, opened a new log
I20240502 01:52:42.218822 11158 ts_tablet_manager.cc:1367] T c96831f69a9b452c811152c9b0de7588 P 34da849acb2045a9a2ef03933592dd08: Time spent bootstrapping tablet: real 0.015s	user 0.000s	sys 0.004s
I20240502 01:52:42.219417 11157 raft_consensus.cc:348] T 930071b72771448d98cbe659ee23c63a P ede1dd598e3541138a19a3ea5c052bc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } }
I20240502 01:52:42.219664 11157 raft_consensus.cc:374] T 930071b72771448d98cbe659ee23c63a P ede1dd598e3541138a19a3ea5c052bc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:42.219771 11157 raft_consensus.cc:729] T 930071b72771448d98cbe659ee23c63a P ede1dd598e3541138a19a3ea5c052bc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ede1dd598e3541138a19a3ea5c052bc9, State: Initialized, Role: FOLLOWER
I20240502 01:52:42.219934 11156 tablet_bootstrap.cc:654] T c264070710e8405ab9c2b3fe2cea46ae P fd95bcd4cdcb4cc5966050ffe2cffdff: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:42.220032 11157 consensus_queue.cc:260] T 930071b72771448d98cbe659ee23c63a P ede1dd598e3541138a19a3ea5c052bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } }
I20240502 01:52:42.220552 11158 raft_consensus.cc:348] T c96831f69a9b452c811152c9b0de7588 P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } }
I20240502 01:52:42.220788 11158 raft_consensus.cc:374] T c96831f69a9b452c811152c9b0de7588 P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:42.220914 11158 raft_consensus.cc:729] T c96831f69a9b452c811152c9b0de7588 P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34da849acb2045a9a2ef03933592dd08, State: Initialized, Role: FOLLOWER
I20240502 01:52:42.221199 11153 tablet_bootstrap.cc:654] T c264070710e8405ab9c2b3fe2cea46ae P 7b7a47245f294ce9a264fcc8f6351c0f: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:42.221176 11158 consensus_queue.cc:260] T c96831f69a9b452c811152c9b0de7588 P 34da849acb2045a9a2ef03933592dd08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } }
I20240502 01:52:42.221916 11157 ts_tablet_manager.cc:1398] T 930071b72771448d98cbe659ee23c63a P ede1dd598e3541138a19a3ea5c052bc9: Time spent starting tablet: real 0.003s	user 0.003s	sys 0.000s
I20240502 01:52:42.222777 11158 ts_tablet_manager.cc:1398] T c96831f69a9b452c811152c9b0de7588 P 34da849acb2045a9a2ef03933592dd08: Time spent starting tablet: real 0.004s	user 0.000s	sys 0.003s
I20240502 01:52:42.223094 11158 tablet_bootstrap.cc:492] T c264070710e8405ab9c2b3fe2cea46ae P 34da849acb2045a9a2ef03933592dd08: Bootstrap starting.
I20240502 01:52:42.225229 11158 tablet_bootstrap.cc:654] T c264070710e8405ab9c2b3fe2cea46ae P 34da849acb2045a9a2ef03933592dd08: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:42.225330 11154 tablet_bootstrap.cc:492] T 930071b72771448d98cbe659ee23c63a P a885359a82bd4515913f7b273591f851: No bootstrap required, opened a new log
I20240502 01:52:42.225517 11154 ts_tablet_manager.cc:1367] T 930071b72771448d98cbe659ee23c63a P a885359a82bd4515913f7b273591f851: Time spent bootstrapping tablet: real 0.018s	user 0.005s	sys 0.000s
I20240502 01:52:42.227525 11153 tablet_bootstrap.cc:492] T c264070710e8405ab9c2b3fe2cea46ae P 7b7a47245f294ce9a264fcc8f6351c0f: No bootstrap required, opened a new log
I20240502 01:52:42.227807 11153 ts_tablet_manager.cc:1367] T c264070710e8405ab9c2b3fe2cea46ae P 7b7a47245f294ce9a264fcc8f6351c0f: Time spent bootstrapping tablet: real 0.016s	user 0.006s	sys 0.000s
I20240502 01:52:42.228705 11153 raft_consensus.cc:348] T c264070710e8405ab9c2b3fe2cea46ae P 7b7a47245f294ce9a264fcc8f6351c0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } } peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } }
I20240502 01:52:42.228849 11154 raft_consensus.cc:348] T 930071b72771448d98cbe659ee23c63a P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } }
I20240502 01:52:42.229025 11153 raft_consensus.cc:374] T c264070710e8405ab9c2b3fe2cea46ae P 7b7a47245f294ce9a264fcc8f6351c0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:42.229106 11154 raft_consensus.cc:374] T 930071b72771448d98cbe659ee23c63a P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:42.229175 11153 raft_consensus.cc:729] T c264070710e8405ab9c2b3fe2cea46ae P 7b7a47245f294ce9a264fcc8f6351c0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b7a47245f294ce9a264fcc8f6351c0f, State: Initialized, Role: FOLLOWER
I20240502 01:52:42.229266 11154 raft_consensus.cc:729] T 930071b72771448d98cbe659ee23c63a P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a885359a82bd4515913f7b273591f851, State: Initialized, Role: FOLLOWER
I20240502 01:52:42.229573 11154 consensus_queue.cc:260] T 930071b72771448d98cbe659ee23c63a P a885359a82bd4515913f7b273591f851 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } }
I20240502 01:52:42.229609 11153 consensus_queue.cc:260] T c264070710e8405ab9c2b3fe2cea46ae P 7b7a47245f294ce9a264fcc8f6351c0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } } peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } }
I20240502 01:52:42.229979 11156 tablet_bootstrap.cc:492] T c264070710e8405ab9c2b3fe2cea46ae P fd95bcd4cdcb4cc5966050ffe2cffdff: No bootstrap required, opened a new log
I20240502 01:52:42.229980 11158 tablet_bootstrap.cc:492] T c264070710e8405ab9c2b3fe2cea46ae P 34da849acb2045a9a2ef03933592dd08: No bootstrap required, opened a new log
I20240502 01:52:42.230157 11156 ts_tablet_manager.cc:1367] T c264070710e8405ab9c2b3fe2cea46ae P fd95bcd4cdcb4cc5966050ffe2cffdff: Time spent bootstrapping tablet: real 0.014s	user 0.005s	sys 0.000s
I20240502 01:52:42.230157 11158 ts_tablet_manager.cc:1367] T c264070710e8405ab9c2b3fe2cea46ae P 34da849acb2045a9a2ef03933592dd08: Time spent bootstrapping tablet: real 0.007s	user 0.004s	sys 0.000s
I20240502 01:52:42.230293 11153 ts_tablet_manager.cc:1398] T c264070710e8405ab9c2b3fe2cea46ae P 7b7a47245f294ce9a264fcc8f6351c0f: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 01:52:42.230968 11158 raft_consensus.cc:348] T c264070710e8405ab9c2b3fe2cea46ae P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } } peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } }
I20240502 01:52:42.230971 11156 raft_consensus.cc:348] T c264070710e8405ab9c2b3fe2cea46ae P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } } peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } }
I20240502 01:52:42.231199 11156 raft_consensus.cc:374] T c264070710e8405ab9c2b3fe2cea46ae P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:42.231201 11158 raft_consensus.cc:374] T c264070710e8405ab9c2b3fe2cea46ae P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:42.231276 11156 raft_consensus.cc:729] T c264070710e8405ab9c2b3fe2cea46ae P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd95bcd4cdcb4cc5966050ffe2cffdff, State: Initialized, Role: FOLLOWER
I20240502 01:52:42.231276 11158 raft_consensus.cc:729] T c264070710e8405ab9c2b3fe2cea46ae P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34da849acb2045a9a2ef03933592dd08, State: Initialized, Role: FOLLOWER
I20240502 01:52:42.231523 11156 consensus_queue.cc:260] T c264070710e8405ab9c2b3fe2cea46ae P fd95bcd4cdcb4cc5966050ffe2cffdff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } } peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } }
I20240502 01:52:42.231592 11158 consensus_queue.cc:260] T c264070710e8405ab9c2b3fe2cea46ae P 34da849acb2045a9a2ef03933592dd08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } } peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } }
I20240502 01:52:42.232128 11156 ts_tablet_manager.cc:1398] T c264070710e8405ab9c2b3fe2cea46ae P fd95bcd4cdcb4cc5966050ffe2cffdff: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240502 01:52:42.233965 11158 ts_tablet_manager.cc:1398] T c264070710e8405ab9c2b3fe2cea46ae P 34da849acb2045a9a2ef03933592dd08: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.000s
I20240502 01:52:42.234536 11133 raft_consensus.cc:482] T c264070710e8405ab9c2b3fe2cea46ae P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:42.234766 11154 ts_tablet_manager.cc:1398] T 930071b72771448d98cbe659ee23c63a P a885359a82bd4515913f7b273591f851: Time spent starting tablet: real 0.009s	user 0.003s	sys 0.000s
I20240502 01:52:42.234755 11133 raft_consensus.cc:504] T c264070710e8405ab9c2b3fe2cea46ae P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } } peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } }
I20240502 01:52:42.237075 11133 leader_election.cc:288] T c264070710e8405ab9c2b3fe2cea46ae P 34da849acb2045a9a2ef03933592dd08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b7a47245f294ce9a264fcc8f6351c0f (127.9.176.194:43645), fd95bcd4cdcb4cc5966050ffe2cffdff (127.9.176.198:33039)
I20240502 01:52:42.251227 10245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c264070710e8405ab9c2b3fe2cea46ae" candidate_uuid: "34da849acb2045a9a2ef03933592dd08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" is_pre_election: true
I20240502 01:52:42.251516 10245 raft_consensus.cc:2454] T c264070710e8405ab9c2b3fe2cea46ae P 7b7a47245f294ce9a264fcc8f6351c0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34da849acb2045a9a2ef03933592dd08 in term 0.
I20240502 01:52:42.251916 10780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c264070710e8405ab9c2b3fe2cea46ae" candidate_uuid: "34da849acb2045a9a2ef03933592dd08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" is_pre_election: true
I20240502 01:52:42.252125 10443 leader_election.cc:302] T c264070710e8405ab9c2b3fe2cea46ae P 34da849acb2045a9a2ef03933592dd08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34da849acb2045a9a2ef03933592dd08, 7b7a47245f294ce9a264fcc8f6351c0f; no voters: 
I20240502 01:52:42.252446 10780 raft_consensus.cc:2454] T c264070710e8405ab9c2b3fe2cea46ae P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34da849acb2045a9a2ef03933592dd08 in term 0.
I20240502 01:52:42.252610 11133 raft_consensus.cc:2781] T c264070710e8405ab9c2b3fe2cea46ae P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:42.252739 11133 raft_consensus.cc:482] T c264070710e8405ab9c2b3fe2cea46ae P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:42.252833 11133 raft_consensus.cc:3037] T c264070710e8405ab9c2b3fe2cea46ae P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:42.264184 11133 raft_consensus.cc:504] T c264070710e8405ab9c2b3fe2cea46ae P 34da849acb2045a9a2ef03933592dd08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } } peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } }
I20240502 01:52:42.265270 11133 leader_election.cc:288] T c264070710e8405ab9c2b3fe2cea46ae P 34da849acb2045a9a2ef03933592dd08 [CANDIDATE]: Term 1 election: Requested vote from peers 7b7a47245f294ce9a264fcc8f6351c0f (127.9.176.194:43645), fd95bcd4cdcb4cc5966050ffe2cffdff (127.9.176.198:33039)
I20240502 01:52:42.265936 10780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c264070710e8405ab9c2b3fe2cea46ae" candidate_uuid: "34da849acb2045a9a2ef03933592dd08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff"
I20240502 01:52:42.265969 10245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c264070710e8405ab9c2b3fe2cea46ae" candidate_uuid: "34da849acb2045a9a2ef03933592dd08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b7a47245f294ce9a264fcc8f6351c0f"
I20240502 01:52:42.266180 10780 raft_consensus.cc:3037] T c264070710e8405ab9c2b3fe2cea46ae P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:42.266223 10245 raft_consensus.cc:3037] T c264070710e8405ab9c2b3fe2cea46ae P 7b7a47245f294ce9a264fcc8f6351c0f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:42.268544 10780 raft_consensus.cc:2454] T c264070710e8405ab9c2b3fe2cea46ae P fd95bcd4cdcb4cc5966050ffe2cffdff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34da849acb2045a9a2ef03933592dd08 in term 1.
I20240502 01:52:42.269136 10445 leader_election.cc:302] T c264070710e8405ab9c2b3fe2cea46ae P 34da849acb2045a9a2ef03933592dd08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34da849acb2045a9a2ef03933592dd08, fd95bcd4cdcb4cc5966050ffe2cffdff; no voters: 
I20240502 01:52:42.269634 11133 raft_consensus.cc:2781] T c264070710e8405ab9c2b3fe2cea46ae P 34da849acb2045a9a2ef03933592dd08 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:42.269739 10245 raft_consensus.cc:2454] T c264070710e8405ab9c2b3fe2cea46ae P 7b7a47245f294ce9a264fcc8f6351c0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34da849acb2045a9a2ef03933592dd08 in term 1.
I20240502 01:52:42.270535 11133 raft_consensus.cc:686] T c264070710e8405ab9c2b3fe2cea46ae P 34da849acb2045a9a2ef03933592dd08 [term 1 LEADER]: Becoming Leader. State: Replica: 34da849acb2045a9a2ef03933592dd08, State: Running, Role: LEADER
I20240502 01:52:42.271153 11133 consensus_queue.cc:237] T c264070710e8405ab9c2b3fe2cea46ae P 34da849acb2045a9a2ef03933592dd08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } } peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } }
I20240502 01:52:42.275032  9962 catalog_manager.cc:5484] T c264070710e8405ab9c2b3fe2cea46ae P 34da849acb2045a9a2ef03933592dd08 reported cstate change: term changed from 0 to 1, leader changed from <none> to 34da849acb2045a9a2ef03933592dd08 (127.9.176.196). New cstate: current_term: 1 leader_uuid: "34da849acb2045a9a2ef03933592dd08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } health_report { overall_health: UNKNOWN } } }
I20240502 01:52:42.445156 10902 raft_consensus.cc:482] T 930071b72771448d98cbe659ee23c63a P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:42.445412 10902 raft_consensus.cc:504] T 930071b72771448d98cbe659ee23c63a P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } }
I20240502 01:52:42.446170 10902 leader_election.cc:288] T 930071b72771448d98cbe659ee23c63a P a8c4a518929d4bb3a88bdfb1649493bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a885359a82bd4515913f7b273591f851 (127.9.176.193:38373), ede1dd598e3541138a19a3ea5c052bc9 (127.9.176.195:38701)
I20240502 01:52:42.446774 10112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "930071b72771448d98cbe659ee23c63a" candidate_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a885359a82bd4515913f7b273591f851" is_pre_election: true
I20240502 01:52:42.446820 10378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "930071b72771448d98cbe659ee23c63a" candidate_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ede1dd598e3541138a19a3ea5c052bc9" is_pre_election: true
I20240502 01:52:42.447070 10378 raft_consensus.cc:2454] T 930071b72771448d98cbe659ee23c63a P ede1dd598e3541138a19a3ea5c052bc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8c4a518929d4bb3a88bdfb1649493bd in term 0.
I20240502 01:52:42.447067 10112 raft_consensus.cc:2454] T 930071b72771448d98cbe659ee23c63a P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8c4a518929d4bb3a88bdfb1649493bd in term 0.
I20240502 01:52:42.447124 11133 raft_consensus.cc:482] T c96831f69a9b452c811152c9b0de7588 P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:42.447350 11133 raft_consensus.cc:504] T c96831f69a9b452c811152c9b0de7588 P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } }
I20240502 01:52:42.447628 10579 leader_election.cc:302] T 930071b72771448d98cbe659ee23c63a P a8c4a518929d4bb3a88bdfb1649493bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a8c4a518929d4bb3a88bdfb1649493bd, ede1dd598e3541138a19a3ea5c052bc9; no voters: 
I20240502 01:52:42.447942 10902 raft_consensus.cc:2781] T 930071b72771448d98cbe659ee23c63a P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:42.448055 10902 raft_consensus.cc:482] T 930071b72771448d98cbe659ee23c63a P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:42.448179 10902 raft_consensus.cc:3037] T 930071b72771448d98cbe659ee23c63a P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:42.448266 11133 leader_election.cc:288] T c96831f69a9b452c811152c9b0de7588 P 34da849acb2045a9a2ef03933592dd08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fd95bcd4cdcb4cc5966050ffe2cffdff (127.9.176.198:33039), 7b7a47245f294ce9a264fcc8f6351c0f (127.9.176.194:43645)
I20240502 01:52:42.448856 10246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c96831f69a9b452c811152c9b0de7588" candidate_uuid: "34da849acb2045a9a2ef03933592dd08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" is_pre_election: true
I20240502 01:52:42.448872 10780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c96831f69a9b452c811152c9b0de7588" candidate_uuid: "34da849acb2045a9a2ef03933592dd08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" is_pre_election: true
I20240502 01:52:42.449131 10246 raft_consensus.cc:2454] T c96831f69a9b452c811152c9b0de7588 P 7b7a47245f294ce9a264fcc8f6351c0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34da849acb2045a9a2ef03933592dd08 in term 0.
I20240502 01:52:42.449144 10780 raft_consensus.cc:2454] T c96831f69a9b452c811152c9b0de7588 P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34da849acb2045a9a2ef03933592dd08 in term 0.
I20240502 01:52:42.449657 10443 leader_election.cc:302] T c96831f69a9b452c811152c9b0de7588 P 34da849acb2045a9a2ef03933592dd08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34da849acb2045a9a2ef03933592dd08, 7b7a47245f294ce9a264fcc8f6351c0f; no voters: 
I20240502 01:52:42.450002 11133 raft_consensus.cc:2781] T c96831f69a9b452c811152c9b0de7588 P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:42.450091 11133 raft_consensus.cc:482] T c96831f69a9b452c811152c9b0de7588 P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:42.450197 11133 raft_consensus.cc:3037] T c96831f69a9b452c811152c9b0de7588 P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:42.473328 10902 raft_consensus.cc:504] T 930071b72771448d98cbe659ee23c63a P a8c4a518929d4bb3a88bdfb1649493bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } }
I20240502 01:52:42.473328 11133 raft_consensus.cc:504] T c96831f69a9b452c811152c9b0de7588 P 34da849acb2045a9a2ef03933592dd08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } }
I20240502 01:52:42.474238 11133 leader_election.cc:288] T c96831f69a9b452c811152c9b0de7588 P 34da849acb2045a9a2ef03933592dd08 [CANDIDATE]: Term 1 election: Requested vote from peers fd95bcd4cdcb4cc5966050ffe2cffdff (127.9.176.198:33039), 7b7a47245f294ce9a264fcc8f6351c0f (127.9.176.194:43645)
I20240502 01:52:42.475010 10902 leader_election.cc:288] T 930071b72771448d98cbe659ee23c63a P a8c4a518929d4bb3a88bdfb1649493bd [CANDIDATE]: Term 1 election: Requested vote from peers a885359a82bd4515913f7b273591f851 (127.9.176.193:38373), ede1dd598e3541138a19a3ea5c052bc9 (127.9.176.195:38701)
I20240502 01:52:42.475416 10378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "930071b72771448d98cbe659ee23c63a" candidate_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ede1dd598e3541138a19a3ea5c052bc9"
I20240502 01:52:42.475710 10378 raft_consensus.cc:3037] T 930071b72771448d98cbe659ee23c63a P ede1dd598e3541138a19a3ea5c052bc9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:42.476017 10112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "930071b72771448d98cbe659ee23c63a" candidate_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a885359a82bd4515913f7b273591f851"
I20240502 01:52:42.476236 10112 raft_consensus.cc:3037] T 930071b72771448d98cbe659ee23c63a P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:42.478091 10246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c96831f69a9b452c811152c9b0de7588" candidate_uuid: "34da849acb2045a9a2ef03933592dd08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b7a47245f294ce9a264fcc8f6351c0f"
I20240502 01:52:42.478314 10246 raft_consensus.cc:3037] T c96831f69a9b452c811152c9b0de7588 P 7b7a47245f294ce9a264fcc8f6351c0f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:42.479652 10780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c96831f69a9b452c811152c9b0de7588" candidate_uuid: "34da849acb2045a9a2ef03933592dd08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff"
I20240502 01:52:42.479910 10780 raft_consensus.cc:3037] T c96831f69a9b452c811152c9b0de7588 P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:42.572924 10112 raft_consensus.cc:2454] T 930071b72771448d98cbe659ee23c63a P a885359a82bd4515913f7b273591f851 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8c4a518929d4bb3a88bdfb1649493bd in term 1.
I20240502 01:52:42.572913 10378 raft_consensus.cc:2454] T 930071b72771448d98cbe659ee23c63a P ede1dd598e3541138a19a3ea5c052bc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8c4a518929d4bb3a88bdfb1649493bd in term 1.
I20240502 01:52:42.572932 10246 raft_consensus.cc:2454] T c96831f69a9b452c811152c9b0de7588 P 7b7a47245f294ce9a264fcc8f6351c0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34da849acb2045a9a2ef03933592dd08 in term 1.
I20240502 01:52:42.573684 10443 leader_election.cc:302] T c96831f69a9b452c811152c9b0de7588 P 34da849acb2045a9a2ef03933592dd08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34da849acb2045a9a2ef03933592dd08, 7b7a47245f294ce9a264fcc8f6351c0f; no voters: 
I20240502 01:52:42.573652 10579 leader_election.cc:302] T 930071b72771448d98cbe659ee23c63a P a8c4a518929d4bb3a88bdfb1649493bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a8c4a518929d4bb3a88bdfb1649493bd, ede1dd598e3541138a19a3ea5c052bc9; no voters: 
I20240502 01:52:42.573709 10780 raft_consensus.cc:2454] T c96831f69a9b452c811152c9b0de7588 P fd95bcd4cdcb4cc5966050ffe2cffdff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34da849acb2045a9a2ef03933592dd08 in term 1.
I20240502 01:52:42.574120 11133 raft_consensus.cc:2781] T c96831f69a9b452c811152c9b0de7588 P 34da849acb2045a9a2ef03933592dd08 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:42.574622 10902 raft_consensus.cc:2781] T 930071b72771448d98cbe659ee23c63a P a8c4a518929d4bb3a88bdfb1649493bd [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:42.575910 11133 raft_consensus.cc:686] T c96831f69a9b452c811152c9b0de7588 P 34da849acb2045a9a2ef03933592dd08 [term 1 LEADER]: Becoming Leader. State: Replica: 34da849acb2045a9a2ef03933592dd08, State: Running, Role: LEADER
I20240502 01:52:42.576321 11133 consensus_queue.cc:237] T c96831f69a9b452c811152c9b0de7588 P 34da849acb2045a9a2ef03933592dd08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } }
I20240502 01:52:42.580145 10902 raft_consensus.cc:686] T 930071b72771448d98cbe659ee23c63a P a8c4a518929d4bb3a88bdfb1649493bd [term 1 LEADER]: Becoming Leader. State: Replica: a8c4a518929d4bb3a88bdfb1649493bd, State: Running, Role: LEADER
I20240502 01:52:42.580591 10902 consensus_queue.cc:237] T 930071b72771448d98cbe659ee23c63a P a8c4a518929d4bb3a88bdfb1649493bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } }
I20240502 01:52:42.583550  9962 catalog_manager.cc:5484] T c96831f69a9b452c811152c9b0de7588 P 34da849acb2045a9a2ef03933592dd08 reported cstate change: term changed from 0 to 1, leader changed from <none> to 34da849acb2045a9a2ef03933592dd08 (127.9.176.196). New cstate: current_term: 1 leader_uuid: "34da849acb2045a9a2ef03933592dd08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } health_report { overall_health: HEALTHY } } }
I20240502 01:52:42.587795  9964 catalog_manager.cc:5484] T 930071b72771448d98cbe659ee23c63a P a8c4a518929d4bb3a88bdfb1649493bd reported cstate change: term changed from 0 to 1, leader changed from <none> to a8c4a518929d4bb3a88bdfb1649493bd (127.9.176.197). New cstate: current_term: 1 leader_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } health_report { overall_health: UNKNOWN } } }
I20240502 01:52:42.678602 11133 consensus_queue.cc:1035] T c264070710e8405ab9c2b3fe2cea46ae P 34da849acb2045a9a2ef03933592dd08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:42.684590 11173 consensus_queue.cc:1035] T c264070710e8405ab9c2b3fe2cea46ae P 34da849acb2045a9a2ef03933592dd08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:42.978461 10902 consensus_queue.cc:1035] T 930071b72771448d98cbe659ee23c63a P a8c4a518929d4bb3a88bdfb1649493bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:42.986526 11178 consensus_queue.cc:1035] T 930071b72771448d98cbe659ee23c63a P a8c4a518929d4bb3a88bdfb1649493bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:43.011484 11173 consensus_queue.cc:1035] T c96831f69a9b452c811152c9b0de7588 P 34da849acb2045a9a2ef03933592dd08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:43.013530 11173 consensus_queue.cc:1035] T c96831f69a9b452c811152c9b0de7588 P 34da849acb2045a9a2ef03933592dd08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:52:43.082641 11186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:43.083078 11186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:43.095839 11186 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'

INSERT report
    rows total: 2000
    time total: 236.778 ms
  time per row: 0.118389 ms
I20240502 01:52:43.487234  9965 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56524:
name: "rebalance_test_table_4"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 01:52:43.498135 10623 tablet_service.cc:1467] Processing CreateTablet for tablet 2f803cd8e5bd420c8f0417d92af153b5 (DEFAULT_TABLE table=rebalance_test_table_4 [id=87f19d2fec1c42ef82386505eb89792e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:43.498692 10623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f803cd8e5bd420c8f0417d92af153b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:43.499712 10224 tablet_service.cc:1467] Processing CreateTablet for tablet d31a3838cfbc4f3b9990048fbd340421 (DEFAULT_TABLE table=rebalance_test_table_4 [id=87f19d2fec1c42ef82386505eb89792e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:43.500237 10224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d31a3838cfbc4f3b9990048fbd340421. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:43.500402 10091 tablet_service.cc:1467] Processing CreateTablet for tablet b7218b5b2e2c4e1a862705edd8574f6b (DEFAULT_TABLE table=rebalance_test_table_4 [id=87f19d2fec1c42ef82386505eb89792e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:43.500829 10091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7218b5b2e2c4e1a862705edd8574f6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:43.501960 10358 tablet_service.cc:1467] Processing CreateTablet for tablet 2f803cd8e5bd420c8f0417d92af153b5 (DEFAULT_TABLE table=rebalance_test_table_4 [id=87f19d2fec1c42ef82386505eb89792e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:43.502099 10092 tablet_service.cc:1467] Processing CreateTablet for tablet 2f803cd8e5bd420c8f0417d92af153b5 (DEFAULT_TABLE table=rebalance_test_table_4 [id=87f19d2fec1c42ef82386505eb89792e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:43.502419 10358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f803cd8e5bd420c8f0417d92af153b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:43.502486 10092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f803cd8e5bd420c8f0417d92af153b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:43.504184 10492 tablet_service.cc:1467] Processing CreateTablet for tablet d31a3838cfbc4f3b9990048fbd340421 (DEFAULT_TABLE table=rebalance_test_table_4 [id=87f19d2fec1c42ef82386505eb89792e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:43.504617 10492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d31a3838cfbc4f3b9990048fbd340421. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:43.508281 10758 tablet_service.cc:1467] Processing CreateTablet for tablet d31a3838cfbc4f3b9990048fbd340421 (DEFAULT_TABLE table=rebalance_test_table_4 [id=87f19d2fec1c42ef82386505eb89792e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:43.508773 10758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d31a3838cfbc4f3b9990048fbd340421. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:43.509573 10624 tablet_service.cc:1467] Processing CreateTablet for tablet b7218b5b2e2c4e1a862705edd8574f6b (DEFAULT_TABLE table=rebalance_test_table_4 [id=87f19d2fec1c42ef82386505eb89792e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:43.510143 10624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7218b5b2e2c4e1a862705edd8574f6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:43.507560 10357 tablet_service.cc:1467] Processing CreateTablet for tablet b7218b5b2e2c4e1a862705edd8574f6b (DEFAULT_TABLE table=rebalance_test_table_4 [id=87f19d2fec1c42ef82386505eb89792e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:43.511912 11247 tablet_bootstrap.cc:492] T d31a3838cfbc4f3b9990048fbd340421 P 7b7a47245f294ce9a264fcc8f6351c0f: Bootstrap starting.
I20240502 01:52:43.512185 10357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7218b5b2e2c4e1a862705edd8574f6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:43.513796 11247 tablet_bootstrap.cc:654] T d31a3838cfbc4f3b9990048fbd340421 P 7b7a47245f294ce9a264fcc8f6351c0f: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:43.515508 11249 tablet_bootstrap.cc:492] T 2f803cd8e5bd420c8f0417d92af153b5 P a8c4a518929d4bb3a88bdfb1649493bd: Bootstrap starting.
I20240502 01:52:43.517463 11249 tablet_bootstrap.cc:654] T 2f803cd8e5bd420c8f0417d92af153b5 P a8c4a518929d4bb3a88bdfb1649493bd: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:43.534759 11252 tablet_bootstrap.cc:492] T b7218b5b2e2c4e1a862705edd8574f6b P a885359a82bd4515913f7b273591f851: Bootstrap starting.
I20240502 01:52:43.540665 11247 tablet_bootstrap.cc:492] T d31a3838cfbc4f3b9990048fbd340421 P 7b7a47245f294ce9a264fcc8f6351c0f: No bootstrap required, opened a new log
I20240502 01:52:43.540663 11252 tablet_bootstrap.cc:654] T b7218b5b2e2c4e1a862705edd8574f6b P a885359a82bd4515913f7b273591f851: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:43.540990 11247 ts_tablet_manager.cc:1367] T d31a3838cfbc4f3b9990048fbd340421 P 7b7a47245f294ce9a264fcc8f6351c0f: Time spent bootstrapping tablet: real 0.029s	user 0.004s	sys 0.011s
I20240502 01:52:43.542129 11247 raft_consensus.cc:348] T d31a3838cfbc4f3b9990048fbd340421 P 7b7a47245f294ce9a264fcc8f6351c0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } } peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } }
I20240502 01:52:43.542711 11247 raft_consensus.cc:374] T d31a3838cfbc4f3b9990048fbd340421 P 7b7a47245f294ce9a264fcc8f6351c0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:43.542882 11247 raft_consensus.cc:729] T d31a3838cfbc4f3b9990048fbd340421 P 7b7a47245f294ce9a264fcc8f6351c0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b7a47245f294ce9a264fcc8f6351c0f, State: Initialized, Role: FOLLOWER
I20240502 01:52:43.543028 11254 tablet_bootstrap.cc:492] T d31a3838cfbc4f3b9990048fbd340421 P fd95bcd4cdcb4cc5966050ffe2cffdff: Bootstrap starting.
I20240502 01:52:43.543272 11247 consensus_queue.cc:260] T d31a3838cfbc4f3b9990048fbd340421 P 7b7a47245f294ce9a264fcc8f6351c0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } } peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } }
I20240502 01:52:43.552613 11254 tablet_bootstrap.cc:654] T d31a3838cfbc4f3b9990048fbd340421 P fd95bcd4cdcb4cc5966050ffe2cffdff: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:43.556200 11257 tablet_bootstrap.cc:492] T d31a3838cfbc4f3b9990048fbd340421 P 34da849acb2045a9a2ef03933592dd08: Bootstrap starting.
I20240502 01:52:43.557009 11248 tablet_bootstrap.cc:492] T 2f803cd8e5bd420c8f0417d92af153b5 P ede1dd598e3541138a19a3ea5c052bc9: Bootstrap starting.
I20240502 01:52:43.557180 11247 ts_tablet_manager.cc:1398] T d31a3838cfbc4f3b9990048fbd340421 P 7b7a47245f294ce9a264fcc8f6351c0f: Time spent starting tablet: real 0.016s	user 0.006s	sys 0.002s
I20240502 01:52:43.559199 11257 tablet_bootstrap.cc:654] T d31a3838cfbc4f3b9990048fbd340421 P 34da849acb2045a9a2ef03933592dd08: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:43.559875 11254 tablet_bootstrap.cc:492] T d31a3838cfbc4f3b9990048fbd340421 P fd95bcd4cdcb4cc5966050ffe2cffdff: No bootstrap required, opened a new log
I20240502 01:52:43.560612 11254 ts_tablet_manager.cc:1367] T d31a3838cfbc4f3b9990048fbd340421 P fd95bcd4cdcb4cc5966050ffe2cffdff: Time spent bootstrapping tablet: real 0.018s	user 0.003s	sys 0.002s
I20240502 01:52:43.561723 11254 raft_consensus.cc:348] T d31a3838cfbc4f3b9990048fbd340421 P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } } peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } }
I20240502 01:52:43.562161 11254 raft_consensus.cc:374] T d31a3838cfbc4f3b9990048fbd340421 P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:43.562331 11254 raft_consensus.cc:729] T d31a3838cfbc4f3b9990048fbd340421 P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd95bcd4cdcb4cc5966050ffe2cffdff, State: Initialized, Role: FOLLOWER
I20240502 01:52:43.562743 11254 consensus_queue.cc:260] T d31a3838cfbc4f3b9990048fbd340421 P fd95bcd4cdcb4cc5966050ffe2cffdff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } } peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } }
I20240502 01:52:43.563939 11248 tablet_bootstrap.cc:654] T 2f803cd8e5bd420c8f0417d92af153b5 P ede1dd598e3541138a19a3ea5c052bc9: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:43.565071 11249 tablet_bootstrap.cc:492] T 2f803cd8e5bd420c8f0417d92af153b5 P a8c4a518929d4bb3a88bdfb1649493bd: No bootstrap required, opened a new log
I20240502 01:52:43.565223 11249 ts_tablet_manager.cc:1367] T 2f803cd8e5bd420c8f0417d92af153b5 P a8c4a518929d4bb3a88bdfb1649493bd: Time spent bootstrapping tablet: real 0.050s	user 0.005s	sys 0.015s
I20240502 01:52:43.567348 11249 raft_consensus.cc:348] T 2f803cd8e5bd420c8f0417d92af153b5 P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } }
I20240502 01:52:43.567582 11249 raft_consensus.cc:374] T 2f803cd8e5bd420c8f0417d92af153b5 P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:43.567670 11249 raft_consensus.cc:729] T 2f803cd8e5bd420c8f0417d92af153b5 P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8c4a518929d4bb3a88bdfb1649493bd, State: Initialized, Role: FOLLOWER
I20240502 01:52:43.567942 11249 consensus_queue.cc:260] T 2f803cd8e5bd420c8f0417d92af153b5 P a8c4a518929d4bb3a88bdfb1649493bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } }
I20240502 01:52:43.568477 11249 ts_tablet_manager.cc:1398] T 2f803cd8e5bd420c8f0417d92af153b5 P a8c4a518929d4bb3a88bdfb1649493bd: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.002s
I20240502 01:52:43.568764 11249 tablet_bootstrap.cc:492] T b7218b5b2e2c4e1a862705edd8574f6b P a8c4a518929d4bb3a88bdfb1649493bd: Bootstrap starting.
I20240502 01:52:43.570457 11249 tablet_bootstrap.cc:654] T b7218b5b2e2c4e1a862705edd8574f6b P a8c4a518929d4bb3a88bdfb1649493bd: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:43.573036 11249 tablet_bootstrap.cc:492] T b7218b5b2e2c4e1a862705edd8574f6b P a8c4a518929d4bb3a88bdfb1649493bd: No bootstrap required, opened a new log
I20240502 01:52:43.573210 11249 ts_tablet_manager.cc:1367] T b7218b5b2e2c4e1a862705edd8574f6b P a8c4a518929d4bb3a88bdfb1649493bd: Time spent bootstrapping tablet: real 0.004s	user 0.000s	sys 0.003s
I20240502 01:52:43.574055 11249 raft_consensus.cc:348] T b7218b5b2e2c4e1a862705edd8574f6b P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } }
I20240502 01:52:43.574311 11249 raft_consensus.cc:374] T b7218b5b2e2c4e1a862705edd8574f6b P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:43.574378 11249 raft_consensus.cc:729] T b7218b5b2e2c4e1a862705edd8574f6b P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8c4a518929d4bb3a88bdfb1649493bd, State: Initialized, Role: FOLLOWER
I20240502 01:52:43.574618 11249 consensus_queue.cc:260] T b7218b5b2e2c4e1a862705edd8574f6b P a8c4a518929d4bb3a88bdfb1649493bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } }
I20240502 01:52:43.575193 11249 ts_tablet_manager.cc:1398] T b7218b5b2e2c4e1a862705edd8574f6b P a8c4a518929d4bb3a88bdfb1649493bd: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240502 01:52:43.575732 11252 tablet_bootstrap.cc:492] T b7218b5b2e2c4e1a862705edd8574f6b P a885359a82bd4515913f7b273591f851: No bootstrap required, opened a new log
I20240502 01:52:43.575986 11252 ts_tablet_manager.cc:1367] T b7218b5b2e2c4e1a862705edd8574f6b P a885359a82bd4515913f7b273591f851: Time spent bootstrapping tablet: real 0.041s	user 0.008s	sys 0.014s
I20240502 01:52:43.576931 11254 ts_tablet_manager.cc:1398] T d31a3838cfbc4f3b9990048fbd340421 P fd95bcd4cdcb4cc5966050ffe2cffdff: Time spent starting tablet: real 0.016s	user 0.002s	sys 0.000s
I20240502 01:52:43.578055 11248 tablet_bootstrap.cc:492] T 2f803cd8e5bd420c8f0417d92af153b5 P ede1dd598e3541138a19a3ea5c052bc9: No bootstrap required, opened a new log
I20240502 01:52:43.578272 11248 ts_tablet_manager.cc:1367] T 2f803cd8e5bd420c8f0417d92af153b5 P ede1dd598e3541138a19a3ea5c052bc9: Time spent bootstrapping tablet: real 0.021s	user 0.004s	sys 0.000s
I20240502 01:52:43.578351 11257 tablet_bootstrap.cc:492] T d31a3838cfbc4f3b9990048fbd340421 P 34da849acb2045a9a2ef03933592dd08: No bootstrap required, opened a new log
I20240502 01:52:43.578305 11252 raft_consensus.cc:348] T b7218b5b2e2c4e1a862705edd8574f6b P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } }
I20240502 01:52:43.578541 11257 ts_tablet_manager.cc:1367] T d31a3838cfbc4f3b9990048fbd340421 P 34da849acb2045a9a2ef03933592dd08: Time spent bootstrapping tablet: real 0.022s	user 0.007s	sys 0.000s
I20240502 01:52:43.578570 11252 raft_consensus.cc:374] T b7218b5b2e2c4e1a862705edd8574f6b P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:43.578732 11252 raft_consensus.cc:729] T b7218b5b2e2c4e1a862705edd8574f6b P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a885359a82bd4515913f7b273591f851, State: Initialized, Role: FOLLOWER
I20240502 01:52:43.579067 11252 consensus_queue.cc:260] T b7218b5b2e2c4e1a862705edd8574f6b P a885359a82bd4515913f7b273591f851 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } }
I20240502 01:52:43.579322 11257 raft_consensus.cc:348] T d31a3838cfbc4f3b9990048fbd340421 P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } } peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } }
I20240502 01:52:43.579546 11257 raft_consensus.cc:374] T d31a3838cfbc4f3b9990048fbd340421 P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:43.579676 11257 raft_consensus.cc:729] T d31a3838cfbc4f3b9990048fbd340421 P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34da849acb2045a9a2ef03933592dd08, State: Initialized, Role: FOLLOWER
I20240502 01:52:43.579923 11257 consensus_queue.cc:260] T d31a3838cfbc4f3b9990048fbd340421 P 34da849acb2045a9a2ef03933592dd08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } } peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } }
I20240502 01:52:43.580526 11257 ts_tablet_manager.cc:1398] T d31a3838cfbc4f3b9990048fbd340421 P 34da849acb2045a9a2ef03933592dd08: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240502 01:52:43.580108 11252 ts_tablet_manager.cc:1398] T b7218b5b2e2c4e1a862705edd8574f6b P a885359a82bd4515913f7b273591f851: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.000s
I20240502 01:52:43.582274 11252 tablet_bootstrap.cc:492] T 2f803cd8e5bd420c8f0417d92af153b5 P a885359a82bd4515913f7b273591f851: Bootstrap starting.
I20240502 01:52:43.584323 11252 tablet_bootstrap.cc:654] T 2f803cd8e5bd420c8f0417d92af153b5 P a885359a82bd4515913f7b273591f851: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:43.579319 11248 raft_consensus.cc:348] T 2f803cd8e5bd420c8f0417d92af153b5 P ede1dd598e3541138a19a3ea5c052bc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } }
I20240502 01:52:43.588801 11248 raft_consensus.cc:374] T 2f803cd8e5bd420c8f0417d92af153b5 P ede1dd598e3541138a19a3ea5c052bc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:43.588979 11248 raft_consensus.cc:729] T 2f803cd8e5bd420c8f0417d92af153b5 P ede1dd598e3541138a19a3ea5c052bc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ede1dd598e3541138a19a3ea5c052bc9, State: Initialized, Role: FOLLOWER
I20240502 01:52:43.589305 11248 consensus_queue.cc:260] T 2f803cd8e5bd420c8f0417d92af153b5 P ede1dd598e3541138a19a3ea5c052bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } }
I20240502 01:52:43.590116 11252 tablet_bootstrap.cc:492] T 2f803cd8e5bd420c8f0417d92af153b5 P a885359a82bd4515913f7b273591f851: No bootstrap required, opened a new log
I20240502 01:52:43.590289 11252 ts_tablet_manager.cc:1367] T 2f803cd8e5bd420c8f0417d92af153b5 P a885359a82bd4515913f7b273591f851: Time spent bootstrapping tablet: real 0.008s	user 0.005s	sys 0.000s
I20240502 01:52:43.591194 11252 raft_consensus.cc:348] T 2f803cd8e5bd420c8f0417d92af153b5 P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } }
I20240502 01:52:43.591440 11252 raft_consensus.cc:374] T 2f803cd8e5bd420c8f0417d92af153b5 P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:43.591506 11252 raft_consensus.cc:729] T 2f803cd8e5bd420c8f0417d92af153b5 P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a885359a82bd4515913f7b273591f851, State: Initialized, Role: FOLLOWER
I20240502 01:52:43.592343 11252 consensus_queue.cc:260] T 2f803cd8e5bd420c8f0417d92af153b5 P a885359a82bd4515913f7b273591f851 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } }
I20240502 01:52:43.595217 11252 ts_tablet_manager.cc:1398] T 2f803cd8e5bd420c8f0417d92af153b5 P a885359a82bd4515913f7b273591f851: Time spent starting tablet: real 0.005s	user 0.003s	sys 0.000s
I20240502 01:52:43.598228 11248 ts_tablet_manager.cc:1398] T 2f803cd8e5bd420c8f0417d92af153b5 P ede1dd598e3541138a19a3ea5c052bc9: Time spent starting tablet: real 0.020s	user 0.002s	sys 0.000s
I20240502 01:52:43.598592 11248 tablet_bootstrap.cc:492] T b7218b5b2e2c4e1a862705edd8574f6b P ede1dd598e3541138a19a3ea5c052bc9: Bootstrap starting.
I20240502 01:52:43.601107 11248 tablet_bootstrap.cc:654] T b7218b5b2e2c4e1a862705edd8574f6b P ede1dd598e3541138a19a3ea5c052bc9: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:43.607683 11248 tablet_bootstrap.cc:492] T b7218b5b2e2c4e1a862705edd8574f6b P ede1dd598e3541138a19a3ea5c052bc9: No bootstrap required, opened a new log
I20240502 01:52:43.607892 11248 ts_tablet_manager.cc:1367] T b7218b5b2e2c4e1a862705edd8574f6b P ede1dd598e3541138a19a3ea5c052bc9: Time spent bootstrapping tablet: real 0.009s	user 0.002s	sys 0.005s
I20240502 01:52:43.608582 11248 raft_consensus.cc:348] T b7218b5b2e2c4e1a862705edd8574f6b P ede1dd598e3541138a19a3ea5c052bc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } }
I20240502 01:52:43.608777 11248 raft_consensus.cc:374] T b7218b5b2e2c4e1a862705edd8574f6b P ede1dd598e3541138a19a3ea5c052bc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:43.608867 11248 raft_consensus.cc:729] T b7218b5b2e2c4e1a862705edd8574f6b P ede1dd598e3541138a19a3ea5c052bc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ede1dd598e3541138a19a3ea5c052bc9, State: Initialized, Role: FOLLOWER
I20240502 01:52:43.609114 11248 consensus_queue.cc:260] T b7218b5b2e2c4e1a862705edd8574f6b P ede1dd598e3541138a19a3ea5c052bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } }
I20240502 01:52:43.609741 11248 ts_tablet_manager.cc:1398] T b7218b5b2e2c4e1a862705edd8574f6b P ede1dd598e3541138a19a3ea5c052bc9: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 01:52:43.646005 10942 raft_consensus.cc:482] T 2f803cd8e5bd420c8f0417d92af153b5 P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:43.646297 10942 raft_consensus.cc:504] T 2f803cd8e5bd420c8f0417d92af153b5 P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } }
I20240502 01:52:43.647101 10942 leader_election.cc:288] T 2f803cd8e5bd420c8f0417d92af153b5 P a885359a82bd4515913f7b273591f851 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a8c4a518929d4bb3a88bdfb1649493bd (127.9.176.197:44061), ede1dd598e3541138a19a3ea5c052bc9 (127.9.176.195:38701)
I20240502 01:52:43.647677 10378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f803cd8e5bd420c8f0417d92af153b5" candidate_uuid: "a885359a82bd4515913f7b273591f851" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ede1dd598e3541138a19a3ea5c052bc9" is_pre_election: true
I20240502 01:52:43.647655 10644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f803cd8e5bd420c8f0417d92af153b5" candidate_uuid: "a885359a82bd4515913f7b273591f851" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" is_pre_election: true
I20240502 01:52:43.647926 10378 raft_consensus.cc:2454] T 2f803cd8e5bd420c8f0417d92af153b5 P ede1dd598e3541138a19a3ea5c052bc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a885359a82bd4515913f7b273591f851 in term 0.
I20240502 01:52:43.647949 10644 raft_consensus.cc:2454] T 2f803cd8e5bd420c8f0417d92af153b5 P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a885359a82bd4515913f7b273591f851 in term 0.
I20240502 01:52:43.648455 10045 leader_election.cc:302] T 2f803cd8e5bd420c8f0417d92af153b5 P a885359a82bd4515913f7b273591f851 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a885359a82bd4515913f7b273591f851, ede1dd598e3541138a19a3ea5c052bc9; no voters: 
I20240502 01:52:43.648780 10942 raft_consensus.cc:2781] T 2f803cd8e5bd420c8f0417d92af153b5 P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:43.648905 10942 raft_consensus.cc:482] T 2f803cd8e5bd420c8f0417d92af153b5 P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:43.649015 10942 raft_consensus.cc:3037] T 2f803cd8e5bd420c8f0417d92af153b5 P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:43.670819 11176 raft_consensus.cc:482] T b7218b5b2e2c4e1a862705edd8574f6b P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:43.670897 11263 raft_consensus.cc:482] T b7218b5b2e2c4e1a862705edd8574f6b P ede1dd598e3541138a19a3ea5c052bc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:43.671020 11176 raft_consensus.cc:504] T b7218b5b2e2c4e1a862705edd8574f6b P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } }
I20240502 01:52:43.671090 11263 raft_consensus.cc:504] T b7218b5b2e2c4e1a862705edd8574f6b P ede1dd598e3541138a19a3ea5c052bc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } }
I20240502 01:52:43.671872 11176 leader_election.cc:288] T b7218b5b2e2c4e1a862705edd8574f6b P a8c4a518929d4bb3a88bdfb1649493bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ede1dd598e3541138a19a3ea5c052bc9 (127.9.176.195:38701), a885359a82bd4515913f7b273591f851 (127.9.176.193:38373)
I20240502 01:52:43.672554 10112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7218b5b2e2c4e1a862705edd8574f6b" candidate_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a885359a82bd4515913f7b273591f851" is_pre_election: true
I20240502 01:52:43.672621 10378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7218b5b2e2c4e1a862705edd8574f6b" candidate_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ede1dd598e3541138a19a3ea5c052bc9" is_pre_election: true
I20240502 01:52:43.672842 10378 raft_consensus.cc:2454] T b7218b5b2e2c4e1a862705edd8574f6b P ede1dd598e3541138a19a3ea5c052bc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8c4a518929d4bb3a88bdfb1649493bd in term 0.
I20240502 01:52:43.672860 10112 raft_consensus.cc:2454] T b7218b5b2e2c4e1a862705edd8574f6b P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8c4a518929d4bb3a88bdfb1649493bd in term 0.
I20240502 01:52:43.673008 11263 leader_election.cc:288] T b7218b5b2e2c4e1a862705edd8574f6b P ede1dd598e3541138a19a3ea5c052bc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a8c4a518929d4bb3a88bdfb1649493bd (127.9.176.197:44061), a885359a82bd4515913f7b273591f851 (127.9.176.193:38373)
I20240502 01:52:43.673501 10579 leader_election.cc:302] T b7218b5b2e2c4e1a862705edd8574f6b P a8c4a518929d4bb3a88bdfb1649493bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a8c4a518929d4bb3a88bdfb1649493bd, ede1dd598e3541138a19a3ea5c052bc9; no voters: 
I20240502 01:52:43.673954 11176 raft_consensus.cc:2781] T b7218b5b2e2c4e1a862705edd8574f6b P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:43.674082 11176 raft_consensus.cc:482] T b7218b5b2e2c4e1a862705edd8574f6b P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:43.674189 11176 raft_consensus.cc:3037] T b7218b5b2e2c4e1a862705edd8574f6b P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:43.683501 10644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7218b5b2e2c4e1a862705edd8574f6b" candidate_uuid: "ede1dd598e3541138a19a3ea5c052bc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" is_pre_election: true
I20240502 01:52:43.686405 10112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7218b5b2e2c4e1a862705edd8574f6b" candidate_uuid: "ede1dd598e3541138a19a3ea5c052bc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a885359a82bd4515913f7b273591f851" is_pre_election: true
I20240502 01:52:43.686699 10112 raft_consensus.cc:2454] T b7218b5b2e2c4e1a862705edd8574f6b P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ede1dd598e3541138a19a3ea5c052bc9 in term 0.
I20240502 01:52:43.687220 10313 leader_election.cc:302] T b7218b5b2e2c4e1a862705edd8574f6b P ede1dd598e3541138a19a3ea5c052bc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a885359a82bd4515913f7b273591f851, ede1dd598e3541138a19a3ea5c052bc9; no voters: 
I20240502 01:52:43.687671 11263 raft_consensus.cc:2781] T b7218b5b2e2c4e1a862705edd8574f6b P ede1dd598e3541138a19a3ea5c052bc9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:43.687820 11263 raft_consensus.cc:482] T b7218b5b2e2c4e1a862705edd8574f6b P ede1dd598e3541138a19a3ea5c052bc9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:43.687937 11263 raft_consensus.cc:3037] T b7218b5b2e2c4e1a862705edd8574f6b P ede1dd598e3541138a19a3ea5c052bc9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:43.699316 11263 raft_consensus.cc:504] T b7218b5b2e2c4e1a862705edd8574f6b P ede1dd598e3541138a19a3ea5c052bc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } }
I20240502 01:52:43.699316 11176 raft_consensus.cc:504] T b7218b5b2e2c4e1a862705edd8574f6b P a8c4a518929d4bb3a88bdfb1649493bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } }
I20240502 01:52:43.699340 10942 raft_consensus.cc:504] T 2f803cd8e5bd420c8f0417d92af153b5 P a885359a82bd4515913f7b273591f851 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } }
I20240502 01:52:43.700227 11176 leader_election.cc:288] T b7218b5b2e2c4e1a862705edd8574f6b P a8c4a518929d4bb3a88bdfb1649493bd [CANDIDATE]: Term 1 election: Requested vote from peers ede1dd598e3541138a19a3ea5c052bc9 (127.9.176.195:38701), a885359a82bd4515913f7b273591f851 (127.9.176.193:38373)
I20240502 01:52:43.700227 11263 leader_election.cc:288] T b7218b5b2e2c4e1a862705edd8574f6b P ede1dd598e3541138a19a3ea5c052bc9 [CANDIDATE]: Term 1 election: Requested vote from peers a8c4a518929d4bb3a88bdfb1649493bd (127.9.176.197:44061), a885359a82bd4515913f7b273591f851 (127.9.176.193:38373)
I20240502 01:52:43.700299 10942 leader_election.cc:288] T 2f803cd8e5bd420c8f0417d92af153b5 P a885359a82bd4515913f7b273591f851 [CANDIDATE]: Term 1 election: Requested vote from peers a8c4a518929d4bb3a88bdfb1649493bd (127.9.176.197:44061), ede1dd598e3541138a19a3ea5c052bc9 (127.9.176.195:38701)
I20240502 01:52:43.700845 10644 raft_consensus.cc:2379] T b7218b5b2e2c4e1a862705edd8574f6b P a8c4a518929d4bb3a88bdfb1649493bd [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ede1dd598e3541138a19a3ea5c052bc9 in current term 1: Already voted for candidate a8c4a518929d4bb3a88bdfb1649493bd in this term.
I20240502 01:52:43.701185 10643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f803cd8e5bd420c8f0417d92af153b5" candidate_uuid: "a885359a82bd4515913f7b273591f851" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8c4a518929d4bb3a88bdfb1649493bd"
I20240502 01:52:43.701402 10643 raft_consensus.cc:3037] T 2f803cd8e5bd420c8f0417d92af153b5 P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:43.701664 10377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f803cd8e5bd420c8f0417d92af153b5" candidate_uuid: "a885359a82bd4515913f7b273591f851" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ede1dd598e3541138a19a3ea5c052bc9"
I20240502 01:52:43.701898 10377 raft_consensus.cc:3037] T 2f803cd8e5bd420c8f0417d92af153b5 P ede1dd598e3541138a19a3ea5c052bc9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:43.702275 10378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7218b5b2e2c4e1a862705edd8574f6b" candidate_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ede1dd598e3541138a19a3ea5c052bc9"
I20240502 01:52:43.702584 10378 raft_consensus.cc:2379] T b7218b5b2e2c4e1a862705edd8574f6b P ede1dd598e3541138a19a3ea5c052bc9 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a8c4a518929d4bb3a88bdfb1649493bd in current term 1: Already voted for candidate ede1dd598e3541138a19a3ea5c052bc9 in this term.
I20240502 01:52:43.703169 10111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7218b5b2e2c4e1a862705edd8574f6b" candidate_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a885359a82bd4515913f7b273591f851"
I20240502 01:52:43.703435 10111 raft_consensus.cc:3037] T b7218b5b2e2c4e1a862705edd8574f6b P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:43.703480 10112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7218b5b2e2c4e1a862705edd8574f6b" candidate_uuid: "ede1dd598e3541138a19a3ea5c052bc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a885359a82bd4515913f7b273591f851"
I20240502 01:52:43.703992 10644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7218b5b2e2c4e1a862705edd8574f6b" candidate_uuid: "ede1dd598e3541138a19a3ea5c052bc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8c4a518929d4bb3a88bdfb1649493bd"
I20240502 01:52:43.704250 10644 raft_consensus.cc:2379] T b7218b5b2e2c4e1a862705edd8574f6b P a8c4a518929d4bb3a88bdfb1649493bd [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ede1dd598e3541138a19a3ea5c052bc9 in current term 1: Already voted for candidate a8c4a518929d4bb3a88bdfb1649493bd in this term.
I20240502 01:52:43.704780 10313 leader_election.cc:302] T b7218b5b2e2c4e1a862705edd8574f6b P ede1dd598e3541138a19a3ea5c052bc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ede1dd598e3541138a19a3ea5c052bc9; no voters: a885359a82bd4515913f7b273591f851, a8c4a518929d4bb3a88bdfb1649493bd
I20240502 01:52:43.705175 11263 raft_consensus.cc:2726] T b7218b5b2e2c4e1a862705edd8574f6b P ede1dd598e3541138a19a3ea5c052bc9 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240502 01:52:43.725348 10377 raft_consensus.cc:2454] T 2f803cd8e5bd420c8f0417d92af153b5 P ede1dd598e3541138a19a3ea5c052bc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a885359a82bd4515913f7b273591f851 in term 1.
I20240502 01:52:43.725348 10643 raft_consensus.cc:2454] T 2f803cd8e5bd420c8f0417d92af153b5 P a8c4a518929d4bb3a88bdfb1649493bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a885359a82bd4515913f7b273591f851 in term 1.
I20240502 01:52:43.725373 10111 raft_consensus.cc:2454] T b7218b5b2e2c4e1a862705edd8574f6b P a885359a82bd4515913f7b273591f851 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8c4a518929d4bb3a88bdfb1649493bd in term 1.
I20240502 01:52:43.725783 11256 raft_consensus.cc:482] T d31a3838cfbc4f3b9990048fbd340421 P 7b7a47245f294ce9a264fcc8f6351c0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:43.725988 11256 raft_consensus.cc:504] T d31a3838cfbc4f3b9990048fbd340421 P 7b7a47245f294ce9a264fcc8f6351c0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } } peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } }
I20240502 01:52:43.726251 10045 leader_election.cc:302] T 2f803cd8e5bd420c8f0417d92af153b5 P a885359a82bd4515913f7b273591f851 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a885359a82bd4515913f7b273591f851, ede1dd598e3541138a19a3ea5c052bc9; no voters: 
I20240502 01:52:43.726325 10580 leader_election.cc:302] T b7218b5b2e2c4e1a862705edd8574f6b P a8c4a518929d4bb3a88bdfb1649493bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a885359a82bd4515913f7b273591f851, a8c4a518929d4bb3a88bdfb1649493bd; no voters: ede1dd598e3541138a19a3ea5c052bc9
I20240502 01:52:43.726615 10942 raft_consensus.cc:2781] T 2f803cd8e5bd420c8f0417d92af153b5 P a885359a82bd4515913f7b273591f851 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:43.726785 10942 raft_consensus.cc:686] T 2f803cd8e5bd420c8f0417d92af153b5 P a885359a82bd4515913f7b273591f851 [term 1 LEADER]: Becoming Leader. State: Replica: a885359a82bd4515913f7b273591f851, State: Running, Role: LEADER
I20240502 01:52:43.726900 11176 raft_consensus.cc:2781] T b7218b5b2e2c4e1a862705edd8574f6b P a8c4a518929d4bb3a88bdfb1649493bd [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:43.727079 11176 raft_consensus.cc:686] T b7218b5b2e2c4e1a862705edd8574f6b P a8c4a518929d4bb3a88bdfb1649493bd [term 1 LEADER]: Becoming Leader. State: Replica: a8c4a518929d4bb3a88bdfb1649493bd, State: Running, Role: LEADER
I20240502 01:52:43.727097 10942 consensus_queue.cc:237] T 2f803cd8e5bd420c8f0417d92af153b5 P a885359a82bd4515913f7b273591f851 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } }
I20240502 01:52:43.727416 11176 consensus_queue.cc:237] T b7218b5b2e2c4e1a862705edd8574f6b P a8c4a518929d4bb3a88bdfb1649493bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } }
I20240502 01:52:43.728317 11256 leader_election.cc:288] T d31a3838cfbc4f3b9990048fbd340421 P 7b7a47245f294ce9a264fcc8f6351c0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fd95bcd4cdcb4cc5966050ffe2cffdff (127.9.176.198:33039), 34da849acb2045a9a2ef03933592dd08 (127.9.176.196:44521)
I20240502 01:52:43.741698 10512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d31a3838cfbc4f3b9990048fbd340421" candidate_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34da849acb2045a9a2ef03933592dd08" is_pre_election: true
I20240502 01:52:43.742004 10512 raft_consensus.cc:2454] T d31a3838cfbc4f3b9990048fbd340421 P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b7a47245f294ce9a264fcc8f6351c0f in term 0.
I20240502 01:52:43.742672 10177 leader_election.cc:302] T d31a3838cfbc4f3b9990048fbd340421 P 7b7a47245f294ce9a264fcc8f6351c0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34da849acb2045a9a2ef03933592dd08, 7b7a47245f294ce9a264fcc8f6351c0f; no voters: 
I20240502 01:52:43.743398 11256 raft_consensus.cc:2781] T d31a3838cfbc4f3b9990048fbd340421 P 7b7a47245f294ce9a264fcc8f6351c0f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:43.743553 11256 raft_consensus.cc:482] T d31a3838cfbc4f3b9990048fbd340421 P 7b7a47245f294ce9a264fcc8f6351c0f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:43.743688 11256 raft_consensus.cc:3037] T d31a3838cfbc4f3b9990048fbd340421 P 7b7a47245f294ce9a264fcc8f6351c0f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:43.744228  9960 catalog_manager.cc:5484] T b7218b5b2e2c4e1a862705edd8574f6b P a8c4a518929d4bb3a88bdfb1649493bd reported cstate change: term changed from 0 to 1, leader changed from <none> to a8c4a518929d4bb3a88bdfb1649493bd (127.9.176.197). New cstate: current_term: 1 leader_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } health_report { overall_health: UNKNOWN } } }
I20240502 01:52:43.746876  9965 catalog_manager.cc:5484] T 2f803cd8e5bd420c8f0417d92af153b5 P a885359a82bd4515913f7b273591f851 reported cstate change: term changed from 0 to 1, leader changed from <none> to a885359a82bd4515913f7b273591f851 (127.9.176.193). New cstate: current_term: 1 leader_uuid: "a885359a82bd4515913f7b273591f851" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } health_report { overall_health: HEALTHY } } }
I20240502 01:52:43.750371 10780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d31a3838cfbc4f3b9990048fbd340421" candidate_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" is_pre_election: true
I20240502 01:52:43.750592 10780 raft_consensus.cc:2454] T d31a3838cfbc4f3b9990048fbd340421 P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b7a47245f294ce9a264fcc8f6351c0f in term 0.
I20240502 01:52:43.831418 11256 raft_consensus.cc:504] T d31a3838cfbc4f3b9990048fbd340421 P 7b7a47245f294ce9a264fcc8f6351c0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } } peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } }
I20240502 01:52:43.832301 11256 leader_election.cc:288] T d31a3838cfbc4f3b9990048fbd340421 P 7b7a47245f294ce9a264fcc8f6351c0f [CANDIDATE]: Term 1 election: Requested vote from peers fd95bcd4cdcb4cc5966050ffe2cffdff (127.9.176.198:33039), 34da849acb2045a9a2ef03933592dd08 (127.9.176.196:44521)
I20240502 01:52:43.832991 10512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d31a3838cfbc4f3b9990048fbd340421" candidate_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34da849acb2045a9a2ef03933592dd08"
I20240502 01:52:43.833005 10780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d31a3838cfbc4f3b9990048fbd340421" candidate_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff"
I20240502 01:52:43.833235 10512 raft_consensus.cc:3037] T d31a3838cfbc4f3b9990048fbd340421 P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:43.833261 10780 raft_consensus.cc:3037] T d31a3838cfbc4f3b9990048fbd340421 P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:43.932304 10512 raft_consensus.cc:2454] T d31a3838cfbc4f3b9990048fbd340421 P 34da849acb2045a9a2ef03933592dd08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b7a47245f294ce9a264fcc8f6351c0f in term 1.
I20240502 01:52:43.932324 10780 raft_consensus.cc:2454] T d31a3838cfbc4f3b9990048fbd340421 P fd95bcd4cdcb4cc5966050ffe2cffdff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b7a47245f294ce9a264fcc8f6351c0f in term 1.
I20240502 01:52:43.933037 10177 leader_election.cc:302] T d31a3838cfbc4f3b9990048fbd340421 P 7b7a47245f294ce9a264fcc8f6351c0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34da849acb2045a9a2ef03933592dd08, 7b7a47245f294ce9a264fcc8f6351c0f; no voters: 
I20240502 01:52:43.933452 11256 raft_consensus.cc:2781] T d31a3838cfbc4f3b9990048fbd340421 P 7b7a47245f294ce9a264fcc8f6351c0f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:43.933878 11256 raft_consensus.cc:686] T d31a3838cfbc4f3b9990048fbd340421 P 7b7a47245f294ce9a264fcc8f6351c0f [term 1 LEADER]: Becoming Leader. State: Replica: 7b7a47245f294ce9a264fcc8f6351c0f, State: Running, Role: LEADER
I20240502 01:52:43.934600 11256 consensus_queue.cc:237] T d31a3838cfbc4f3b9990048fbd340421 P 7b7a47245f294ce9a264fcc8f6351c0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } } peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } }
I20240502 01:52:43.938880  9965 catalog_manager.cc:5484] T d31a3838cfbc4f3b9990048fbd340421 P 7b7a47245f294ce9a264fcc8f6351c0f reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b7a47245f294ce9a264fcc8f6351c0f (127.9.176.194). New cstate: current_term: 1 leader_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } health_report { overall_health: UNKNOWN } } }
I20240502 01:52:44.156126 11176 consensus_queue.cc:1035] T b7218b5b2e2c4e1a862705edd8574f6b P a8c4a518929d4bb3a88bdfb1649493bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:44.163178 11176 consensus_queue.cc:1035] T b7218b5b2e2c4e1a862705edd8574f6b P a8c4a518929d4bb3a88bdfb1649493bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:44.270081 10942 consensus_queue.cc:1035] T 2f803cd8e5bd420c8f0417d92af153b5 P a885359a82bd4515913f7b273591f851 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:44.280459 11285 consensus_queue.cc:1035] T 2f803cd8e5bd420c8f0417d92af153b5 P a885359a82bd4515913f7b273591f851 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20240502 01:52:44.372653 11277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:44.373116 11277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:44.381572 11277 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240502 01:52:44.452685 11309 consensus_queue.cc:1035] T d31a3838cfbc4f3b9990048fbd340421 P 7b7a47245f294ce9a264fcc8f6351c0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:44.471616 11309 consensus_queue.cc:1035] T d31a3838cfbc4f3b9990048fbd340421 P 7b7a47245f294ce9a264fcc8f6351c0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:52:44.779901 10714 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.176.193:38373, user_credentials={real_user=slave}} blocked reactor thread for 62751.7us
W20240502 01:52:44.855598 11275 log.cc:924] Time spent T d31a3838cfbc4f3b9990048fbd340421 P 7b7a47245f294ce9a264fcc8f6351c0f: Append to log took a long time: real 0.072s	user 0.000s	sys 0.000s
W20240502 01:52:44.863220 11311 log.cc:924] Time spent T d31a3838cfbc4f3b9990048fbd340421 P 34da849acb2045a9a2ef03933592dd08: Append to log took a long time: real 0.071s	user 0.000s	sys 0.000s
W20240502 01:52:44.863220 11278 log.cc:924] Time spent T b7218b5b2e2c4e1a862705edd8574f6b P a885359a82bd4515913f7b273591f851: Append to log took a long time: real 0.073s	user 0.000s	sys 0.000s
W20240502 01:52:44.863297 11283 log.cc:924] Time spent T 2f803cd8e5bd420c8f0417d92af153b5 P ede1dd598e3541138a19a3ea5c052bc9: Append to log took a long time: real 0.071s	user 0.000s	sys 0.000s
W20240502 01:52:44.863598 11310 log.cc:924] Time spent T d31a3838cfbc4f3b9990048fbd340421 P fd95bcd4cdcb4cc5966050ffe2cffdff: Append to log took a long time: real 0.078s	user 0.000s	sys 0.000s
W20240502 01:52:44.863727 11269 log.cc:924] Time spent T 2f803cd8e5bd420c8f0417d92af153b5 P a885359a82bd4515913f7b273591f851: Append to log took a long time: real 0.076s	user 0.000s	sys 0.000s
W20240502 01:52:44.863910 11281 log.cc:924] Time spent T b7218b5b2e2c4e1a862705edd8574f6b P ede1dd598e3541138a19a3ea5c052bc9: Append to log took a long time: real 0.087s	user 0.000s	sys 0.000s
W20240502 01:52:44.864219 11271 log.cc:924] Time spent T b7218b5b2e2c4e1a862705edd8574f6b P a8c4a518929d4bb3a88bdfb1649493bd: Append to log took a long time: real 0.091s	user 0.000s	sys 0.000s
W20240502 01:52:44.865171 11286 log.cc:924] Time spent T 2f803cd8e5bd420c8f0417d92af153b5 P a8c4a518929d4bb3a88bdfb1649493bd: Append to log took a long time: real 0.076s	user 0.000s	sys 0.000s

INSERT report
    rows total: 2000
    time total: 440.605 ms
  time per row: 0.220303 ms
I20240502 01:52:44.982689  9965 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56524:
name: "rebalance_test_table_5"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 01:52:44.993934 10758 tablet_service.cc:1467] Processing CreateTablet for tablet f72ee833c1d54106802dd9191e142284 (DEFAULT_TABLE table=rebalance_test_table_5 [id=09e3d0c5544e40c88291523e6ba19c52]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:44.993914 10224 tablet_service.cc:1467] Processing CreateTablet for tablet f72ee833c1d54106802dd9191e142284 (DEFAULT_TABLE table=rebalance_test_table_5 [id=09e3d0c5544e40c88291523e6ba19c52]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:44.994498 10758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f72ee833c1d54106802dd9191e142284. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:44.994505 10224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f72ee833c1d54106802dd9191e142284. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:44.999639 10492 tablet_service.cc:1467] Processing CreateTablet for tablet f72ee833c1d54106802dd9191e142284 (DEFAULT_TABLE table=rebalance_test_table_5 [id=09e3d0c5544e40c88291523e6ba19c52]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:45.000144 10492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f72ee833c1d54106802dd9191e142284. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:45.000367 10491 tablet_service.cc:1467] Processing CreateTablet for tablet 1da6b311a749427f9fdd394bec7a033f (DEFAULT_TABLE table=rebalance_test_table_5 [id=09e3d0c5544e40c88291523e6ba19c52]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:45.000735 10491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1da6b311a749427f9fdd394bec7a033f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:45.000461 10624 tablet_service.cc:1467] Processing CreateTablet for tablet 968bb8eb10d44a29b33299898ebe453c (DEFAULT_TABLE table=rebalance_test_table_5 [id=09e3d0c5544e40c88291523e6ba19c52]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:45.001721 10624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 968bb8eb10d44a29b33299898ebe453c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:45.004423 10092 tablet_service.cc:1467] Processing CreateTablet for tablet 968bb8eb10d44a29b33299898ebe453c (DEFAULT_TABLE table=rebalance_test_table_5 [id=09e3d0c5544e40c88291523e6ba19c52]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:45.004659 10225 tablet_service.cc:1467] Processing CreateTablet for tablet 1da6b311a749427f9fdd394bec7a033f (DEFAULT_TABLE table=rebalance_test_table_5 [id=09e3d0c5544e40c88291523e6ba19c52]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:45.004951 10092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 968bb8eb10d44a29b33299898ebe453c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:45.005112 10225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1da6b311a749427f9fdd394bec7a033f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:45.009471 10759 tablet_service.cc:1467] Processing CreateTablet for tablet 1da6b311a749427f9fdd394bec7a033f (DEFAULT_TABLE table=rebalance_test_table_5 [id=09e3d0c5544e40c88291523e6ba19c52]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:45.009480 10358 tablet_service.cc:1467] Processing CreateTablet for tablet 968bb8eb10d44a29b33299898ebe453c (DEFAULT_TABLE table=rebalance_test_table_5 [id=09e3d0c5544e40c88291523e6ba19c52]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:45.009991 10759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1da6b311a749427f9fdd394bec7a033f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:45.009979 10358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 968bb8eb10d44a29b33299898ebe453c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:45.010188 11347 tablet_bootstrap.cc:492] T f72ee833c1d54106802dd9191e142284 P 7b7a47245f294ce9a264fcc8f6351c0f: Bootstrap starting.
I20240502 01:52:45.012008 11347 tablet_bootstrap.cc:654] T f72ee833c1d54106802dd9191e142284 P 7b7a47245f294ce9a264fcc8f6351c0f: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:45.021638 11346 tablet_bootstrap.cc:492] T f72ee833c1d54106802dd9191e142284 P fd95bcd4cdcb4cc5966050ffe2cffdff: Bootstrap starting.
I20240502 01:52:45.029354 11346 tablet_bootstrap.cc:654] T f72ee833c1d54106802dd9191e142284 P fd95bcd4cdcb4cc5966050ffe2cffdff: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:45.038408 11350 tablet_bootstrap.cc:492] T 968bb8eb10d44a29b33299898ebe453c P a8c4a518929d4bb3a88bdfb1649493bd: Bootstrap starting.
I20240502 01:52:45.038889 11352 tablet_bootstrap.cc:492] T 968bb8eb10d44a29b33299898ebe453c P ede1dd598e3541138a19a3ea5c052bc9: Bootstrap starting.
I20240502 01:52:45.040921 11349 tablet_bootstrap.cc:492] T 968bb8eb10d44a29b33299898ebe453c P a885359a82bd4515913f7b273591f851: Bootstrap starting.
I20240502 01:52:45.042440 11351 tablet_bootstrap.cc:492] T f72ee833c1d54106802dd9191e142284 P 34da849acb2045a9a2ef03933592dd08: Bootstrap starting.
I20240502 01:52:45.044565 11350 tablet_bootstrap.cc:654] T 968bb8eb10d44a29b33299898ebe453c P a8c4a518929d4bb3a88bdfb1649493bd: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:45.045516 11352 tablet_bootstrap.cc:654] T 968bb8eb10d44a29b33299898ebe453c P ede1dd598e3541138a19a3ea5c052bc9: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:45.046224 11351 tablet_bootstrap.cc:654] T f72ee833c1d54106802dd9191e142284 P 34da849acb2045a9a2ef03933592dd08: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:45.046265 11349 tablet_bootstrap.cc:654] T 968bb8eb10d44a29b33299898ebe453c P a885359a82bd4515913f7b273591f851: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:45.055439 11346 tablet_bootstrap.cc:492] T f72ee833c1d54106802dd9191e142284 P fd95bcd4cdcb4cc5966050ffe2cffdff: No bootstrap required, opened a new log
I20240502 01:52:45.055892 11346 ts_tablet_manager.cc:1367] T f72ee833c1d54106802dd9191e142284 P fd95bcd4cdcb4cc5966050ffe2cffdff: Time spent bootstrapping tablet: real 0.034s	user 0.004s	sys 0.004s
I20240502 01:52:45.057248 11346 raft_consensus.cc:348] T f72ee833c1d54106802dd9191e142284 P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } }
I20240502 01:52:45.058748 11346 raft_consensus.cc:374] T f72ee833c1d54106802dd9191e142284 P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:45.058992 11349 tablet_bootstrap.cc:492] T 968bb8eb10d44a29b33299898ebe453c P a885359a82bd4515913f7b273591f851: No bootstrap required, opened a new log
I20240502 01:52:45.059051 11346 raft_consensus.cc:729] T f72ee833c1d54106802dd9191e142284 P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd95bcd4cdcb4cc5966050ffe2cffdff, State: Initialized, Role: FOLLOWER
I20240502 01:52:45.059281 11349 ts_tablet_manager.cc:1367] T 968bb8eb10d44a29b33299898ebe453c P a885359a82bd4515913f7b273591f851: Time spent bootstrapping tablet: real 0.019s	user 0.010s	sys 0.000s
I20240502 01:52:45.059428 11346 consensus_queue.cc:260] T f72ee833c1d54106802dd9191e142284 P fd95bcd4cdcb4cc5966050ffe2cffdff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } }
I20240502 01:52:45.060416 11347 tablet_bootstrap.cc:492] T f72ee833c1d54106802dd9191e142284 P 7b7a47245f294ce9a264fcc8f6351c0f: No bootstrap required, opened a new log
I20240502 01:52:45.060359 11349 raft_consensus.cc:348] T 968bb8eb10d44a29b33299898ebe453c P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } }
I20240502 01:52:45.060616 11349 raft_consensus.cc:374] T 968bb8eb10d44a29b33299898ebe453c P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:45.060729 11349 raft_consensus.cc:729] T 968bb8eb10d44a29b33299898ebe453c P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a885359a82bd4515913f7b273591f851, State: Initialized, Role: FOLLOWER
I20240502 01:52:45.060770 11347 ts_tablet_manager.cc:1367] T f72ee833c1d54106802dd9191e142284 P 7b7a47245f294ce9a264fcc8f6351c0f: Time spent bootstrapping tablet: real 0.051s	user 0.019s	sys 0.028s
I20240502 01:52:45.061023 11349 consensus_queue.cc:260] T 968bb8eb10d44a29b33299898ebe453c P a885359a82bd4515913f7b273591f851 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } }
I20240502 01:52:45.061901 11347 raft_consensus.cc:348] T f72ee833c1d54106802dd9191e142284 P 7b7a47245f294ce9a264fcc8f6351c0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } }
I20240502 01:52:45.062211 11347 raft_consensus.cc:374] T f72ee833c1d54106802dd9191e142284 P 7b7a47245f294ce9a264fcc8f6351c0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:45.062364 11347 raft_consensus.cc:729] T f72ee833c1d54106802dd9191e142284 P 7b7a47245f294ce9a264fcc8f6351c0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b7a47245f294ce9a264fcc8f6351c0f, State: Initialized, Role: FOLLOWER
I20240502 01:52:45.062726 11347 consensus_queue.cc:260] T f72ee833c1d54106802dd9191e142284 P 7b7a47245f294ce9a264fcc8f6351c0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } }
I20240502 01:52:45.063447 11347 ts_tablet_manager.cc:1398] T f72ee833c1d54106802dd9191e142284 P 7b7a47245f294ce9a264fcc8f6351c0f: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240502 01:52:45.063809 11347 tablet_bootstrap.cc:492] T 1da6b311a749427f9fdd394bec7a033f P 7b7a47245f294ce9a264fcc8f6351c0f: Bootstrap starting.
I20240502 01:52:45.065861 11346 ts_tablet_manager.cc:1398] T f72ee833c1d54106802dd9191e142284 P fd95bcd4cdcb4cc5966050ffe2cffdff: Time spent starting tablet: real 0.010s	user 0.000s	sys 0.003s
I20240502 01:52:45.066185 11347 tablet_bootstrap.cc:654] T 1da6b311a749427f9fdd394bec7a033f P 7b7a47245f294ce9a264fcc8f6351c0f: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:45.071697 11349 ts_tablet_manager.cc:1398] T 968bb8eb10d44a29b33299898ebe453c P a885359a82bd4515913f7b273591f851: Time spent starting tablet: real 0.012s	user 0.002s	sys 0.000s
I20240502 01:52:45.072536 11346 tablet_bootstrap.cc:492] T 1da6b311a749427f9fdd394bec7a033f P fd95bcd4cdcb4cc5966050ffe2cffdff: Bootstrap starting.
I20240502 01:52:45.077066 11352 tablet_bootstrap.cc:492] T 968bb8eb10d44a29b33299898ebe453c P ede1dd598e3541138a19a3ea5c052bc9: No bootstrap required, opened a new log
I20240502 01:52:45.077360 11352 ts_tablet_manager.cc:1367] T 968bb8eb10d44a29b33299898ebe453c P ede1dd598e3541138a19a3ea5c052bc9: Time spent bootstrapping tablet: real 0.038s	user 0.005s	sys 0.000s
I20240502 01:52:45.077534 11350 tablet_bootstrap.cc:492] T 968bb8eb10d44a29b33299898ebe453c P a8c4a518929d4bb3a88bdfb1649493bd: No bootstrap required, opened a new log
I20240502 01:52:45.077863 11350 ts_tablet_manager.cc:1367] T 968bb8eb10d44a29b33299898ebe453c P a8c4a518929d4bb3a88bdfb1649493bd: Time spent bootstrapping tablet: real 0.040s	user 0.004s	sys 0.002s
I20240502 01:52:45.078271 11352 raft_consensus.cc:348] T 968bb8eb10d44a29b33299898ebe453c P ede1dd598e3541138a19a3ea5c052bc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } }
I20240502 01:52:45.078521 11352 raft_consensus.cc:374] T 968bb8eb10d44a29b33299898ebe453c P ede1dd598e3541138a19a3ea5c052bc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:45.078615 11352 raft_consensus.cc:729] T 968bb8eb10d44a29b33299898ebe453c P ede1dd598e3541138a19a3ea5c052bc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ede1dd598e3541138a19a3ea5c052bc9, State: Initialized, Role: FOLLOWER
I20240502 01:52:45.078863 11352 consensus_queue.cc:260] T 968bb8eb10d44a29b33299898ebe453c P ede1dd598e3541138a19a3ea5c052bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } }
I20240502 01:52:45.079495 11350 raft_consensus.cc:348] T 968bb8eb10d44a29b33299898ebe453c P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } }
I20240502 01:52:45.079970 11350 raft_consensus.cc:374] T 968bb8eb10d44a29b33299898ebe453c P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:45.080147 11350 raft_consensus.cc:729] T 968bb8eb10d44a29b33299898ebe453c P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8c4a518929d4bb3a88bdfb1649493bd, State: Initialized, Role: FOLLOWER
I20240502 01:52:45.081038 11350 consensus_queue.cc:260] T 968bb8eb10d44a29b33299898ebe453c P a8c4a518929d4bb3a88bdfb1649493bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } }
I20240502 01:52:45.082927 11351 tablet_bootstrap.cc:492] T f72ee833c1d54106802dd9191e142284 P 34da849acb2045a9a2ef03933592dd08: No bootstrap required, opened a new log
I20240502 01:52:45.083148 11351 ts_tablet_manager.cc:1367] T f72ee833c1d54106802dd9191e142284 P 34da849acb2045a9a2ef03933592dd08: Time spent bootstrapping tablet: real 0.041s	user 0.002s	sys 0.003s
I20240502 01:52:45.083388 11350 ts_tablet_manager.cc:1398] T 968bb8eb10d44a29b33299898ebe453c P a8c4a518929d4bb3a88bdfb1649493bd: Time spent starting tablet: real 0.005s	user 0.002s	sys 0.001s
I20240502 01:52:45.083828 11332 raft_consensus.cc:482] T f72ee833c1d54106802dd9191e142284 P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:45.084040 11332 raft_consensus.cc:504] T f72ee833c1d54106802dd9191e142284 P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } }
I20240502 01:52:45.084304 11351 raft_consensus.cc:348] T f72ee833c1d54106802dd9191e142284 P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } }
I20240502 01:52:45.084527 11351 raft_consensus.cc:374] T f72ee833c1d54106802dd9191e142284 P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:45.084622 11351 raft_consensus.cc:729] T f72ee833c1d54106802dd9191e142284 P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34da849acb2045a9a2ef03933592dd08, State: Initialized, Role: FOLLOWER
I20240502 01:52:45.084954 11332 leader_election.cc:288] T f72ee833c1d54106802dd9191e142284 P fd95bcd4cdcb4cc5966050ffe2cffdff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b7a47245f294ce9a264fcc8f6351c0f (127.9.176.194:43645), 34da849acb2045a9a2ef03933592dd08 (127.9.176.196:44521)
I20240502 01:52:45.084868 11351 consensus_queue.cc:260] T f72ee833c1d54106802dd9191e142284 P 34da849acb2045a9a2ef03933592dd08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } }
I20240502 01:52:45.086287 10246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f72ee833c1d54106802dd9191e142284" candidate_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" is_pre_election: true
I20240502 01:52:45.086374 10512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f72ee833c1d54106802dd9191e142284" candidate_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34da849acb2045a9a2ef03933592dd08" is_pre_election: true
I20240502 01:52:45.086596 10246 raft_consensus.cc:2454] T f72ee833c1d54106802dd9191e142284 P 7b7a47245f294ce9a264fcc8f6351c0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd95bcd4cdcb4cc5966050ffe2cffdff in term 0.
I20240502 01:52:45.086639 10512 raft_consensus.cc:2454] T f72ee833c1d54106802dd9191e142284 P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd95bcd4cdcb4cc5966050ffe2cffdff in term 0.
I20240502 01:52:45.086419 11346 tablet_bootstrap.cc:654] T 1da6b311a749427f9fdd394bec7a033f P fd95bcd4cdcb4cc5966050ffe2cffdff: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:45.087234 10711 leader_election.cc:302] T f72ee833c1d54106802dd9191e142284 P fd95bcd4cdcb4cc5966050ffe2cffdff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34da849acb2045a9a2ef03933592dd08, fd95bcd4cdcb4cc5966050ffe2cffdff; no voters: 
I20240502 01:52:45.089587 11332 raft_consensus.cc:2781] T f72ee833c1d54106802dd9191e142284 P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:45.089745 11332 raft_consensus.cc:482] T f72ee833c1d54106802dd9191e142284 P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:45.089845 11332 raft_consensus.cc:3037] T f72ee833c1d54106802dd9191e142284 P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:45.089898 11351 ts_tablet_manager.cc:1398] T f72ee833c1d54106802dd9191e142284 P 34da849acb2045a9a2ef03933592dd08: Time spent starting tablet: real 0.007s	user 0.002s	sys 0.000s
I20240502 01:52:45.091521 11351 tablet_bootstrap.cc:492] T 1da6b311a749427f9fdd394bec7a033f P 34da849acb2045a9a2ef03933592dd08: Bootstrap starting.
I20240502 01:52:45.093472 11351 tablet_bootstrap.cc:654] T 1da6b311a749427f9fdd394bec7a033f P 34da849acb2045a9a2ef03933592dd08: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:45.094766 11352 ts_tablet_manager.cc:1398] T 968bb8eb10d44a29b33299898ebe453c P ede1dd598e3541138a19a3ea5c052bc9: Time spent starting tablet: real 0.017s	user 0.003s	sys 0.000s
I20240502 01:52:45.098719 11347 tablet_bootstrap.cc:492] T 1da6b311a749427f9fdd394bec7a033f P 7b7a47245f294ce9a264fcc8f6351c0f: No bootstrap required, opened a new log
I20240502 01:52:45.098942 11347 ts_tablet_manager.cc:1367] T 1da6b311a749427f9fdd394bec7a033f P 7b7a47245f294ce9a264fcc8f6351c0f: Time spent bootstrapping tablet: real 0.035s	user 0.009s	sys 0.010s
I20240502 01:52:45.100054 11347 raft_consensus.cc:348] T 1da6b311a749427f9fdd394bec7a033f P 7b7a47245f294ce9a264fcc8f6351c0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } } peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } }
I20240502 01:52:45.100332 11347 raft_consensus.cc:374] T 1da6b311a749427f9fdd394bec7a033f P 7b7a47245f294ce9a264fcc8f6351c0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:45.100421 11347 raft_consensus.cc:729] T 1da6b311a749427f9fdd394bec7a033f P 7b7a47245f294ce9a264fcc8f6351c0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b7a47245f294ce9a264fcc8f6351c0f, State: Initialized, Role: FOLLOWER
I20240502 01:52:45.100711 11347 consensus_queue.cc:260] T 1da6b311a749427f9fdd394bec7a033f P 7b7a47245f294ce9a264fcc8f6351c0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } } peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } }
I20240502 01:52:45.101284 11347 ts_tablet_manager.cc:1398] T 1da6b311a749427f9fdd394bec7a033f P 7b7a47245f294ce9a264fcc8f6351c0f: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 01:52:45.106437 11346 tablet_bootstrap.cc:492] T 1da6b311a749427f9fdd394bec7a033f P fd95bcd4cdcb4cc5966050ffe2cffdff: No bootstrap required, opened a new log
I20240502 01:52:45.106634 11346 ts_tablet_manager.cc:1367] T 1da6b311a749427f9fdd394bec7a033f P fd95bcd4cdcb4cc5966050ffe2cffdff: Time spent bootstrapping tablet: real 0.039s	user 0.002s	sys 0.004s
I20240502 01:52:45.107282 11346 raft_consensus.cc:348] T 1da6b311a749427f9fdd394bec7a033f P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } } peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } }
I20240502 01:52:45.107479 11346 raft_consensus.cc:374] T 1da6b311a749427f9fdd394bec7a033f P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:45.107553 11346 raft_consensus.cc:729] T 1da6b311a749427f9fdd394bec7a033f P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd95bcd4cdcb4cc5966050ffe2cffdff, State: Initialized, Role: FOLLOWER
I20240502 01:52:45.107846 11346 consensus_queue.cc:260] T 1da6b311a749427f9fdd394bec7a033f P fd95bcd4cdcb4cc5966050ffe2cffdff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } } peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } }
I20240502 01:52:45.108501 11346 ts_tablet_manager.cc:1398] T 1da6b311a749427f9fdd394bec7a033f P fd95bcd4cdcb4cc5966050ffe2cffdff: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240502 01:52:45.116282 11351 tablet_bootstrap.cc:492] T 1da6b311a749427f9fdd394bec7a033f P 34da849acb2045a9a2ef03933592dd08: No bootstrap required, opened a new log
I20240502 01:52:45.116923 11351 ts_tablet_manager.cc:1367] T 1da6b311a749427f9fdd394bec7a033f P 34da849acb2045a9a2ef03933592dd08: Time spent bootstrapping tablet: real 0.025s	user 0.004s	sys 0.003s
I20240502 01:52:45.117772 11351 raft_consensus.cc:348] T 1da6b311a749427f9fdd394bec7a033f P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } } peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } }
I20240502 01:52:45.118036 11351 raft_consensus.cc:374] T 1da6b311a749427f9fdd394bec7a033f P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:45.118110 11351 raft_consensus.cc:729] T 1da6b311a749427f9fdd394bec7a033f P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34da849acb2045a9a2ef03933592dd08, State: Initialized, Role: FOLLOWER
I20240502 01:52:45.118429 11351 consensus_queue.cc:260] T 1da6b311a749427f9fdd394bec7a033f P 34da849acb2045a9a2ef03933592dd08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } } peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } }
I20240502 01:52:45.119052 11351 ts_tablet_manager.cc:1398] T 1da6b311a749427f9fdd394bec7a033f P 34da849acb2045a9a2ef03933592dd08: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240502 01:52:45.127375 11339 raft_consensus.cc:482] T 1da6b311a749427f9fdd394bec7a033f P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:45.127605 11339 raft_consensus.cc:504] T 1da6b311a749427f9fdd394bec7a033f P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } } peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } }
I20240502 01:52:45.128378 11339 leader_election.cc:288] T 1da6b311a749427f9fdd394bec7a033f P fd95bcd4cdcb4cc5966050ffe2cffdff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 34da849acb2045a9a2ef03933592dd08 (127.9.176.196:44521), 7b7a47245f294ce9a264fcc8f6351c0f (127.9.176.194:43645)
I20240502 01:52:45.128876 10512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1da6b311a749427f9fdd394bec7a033f" candidate_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34da849acb2045a9a2ef03933592dd08" is_pre_election: true
I20240502 01:52:45.128978 10246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1da6b311a749427f9fdd394bec7a033f" candidate_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" is_pre_election: true
I20240502 01:52:45.129122 10512 raft_consensus.cc:2454] T 1da6b311a749427f9fdd394bec7a033f P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd95bcd4cdcb4cc5966050ffe2cffdff in term 0.
I20240502 01:52:45.129277 10246 raft_consensus.cc:2454] T 1da6b311a749427f9fdd394bec7a033f P 7b7a47245f294ce9a264fcc8f6351c0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd95bcd4cdcb4cc5966050ffe2cffdff in term 0.
I20240502 01:52:45.129561 10711 leader_election.cc:302] T 1da6b311a749427f9fdd394bec7a033f P fd95bcd4cdcb4cc5966050ffe2cffdff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34da849acb2045a9a2ef03933592dd08, fd95bcd4cdcb4cc5966050ffe2cffdff; no voters: 
I20240502 01:52:45.129845 11339 raft_consensus.cc:2781] T 1da6b311a749427f9fdd394bec7a033f P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:45.129976 11339 raft_consensus.cc:482] T 1da6b311a749427f9fdd394bec7a033f P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:45.130074 11339 raft_consensus.cc:3037] T 1da6b311a749427f9fdd394bec7a033f P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:45.169842 11332 raft_consensus.cc:504] T f72ee833c1d54106802dd9191e142284 P fd95bcd4cdcb4cc5966050ffe2cffdff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } }
I20240502 01:52:45.170738 11332 leader_election.cc:288] T f72ee833c1d54106802dd9191e142284 P fd95bcd4cdcb4cc5966050ffe2cffdff [CANDIDATE]: Term 1 election: Requested vote from peers 7b7a47245f294ce9a264fcc8f6351c0f (127.9.176.194:43645), 34da849acb2045a9a2ef03933592dd08 (127.9.176.196:44521)
I20240502 01:52:45.171548 10246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f72ee833c1d54106802dd9191e142284" candidate_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b7a47245f294ce9a264fcc8f6351c0f"
I20240502 01:52:45.171582 10512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f72ee833c1d54106802dd9191e142284" candidate_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34da849acb2045a9a2ef03933592dd08"
I20240502 01:52:45.171857 10246 raft_consensus.cc:3037] T f72ee833c1d54106802dd9191e142284 P 7b7a47245f294ce9a264fcc8f6351c0f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:45.171885 10512 raft_consensus.cc:3037] T f72ee833c1d54106802dd9191e142284 P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:45.172544 11339 raft_consensus.cc:504] T 1da6b311a749427f9fdd394bec7a033f P fd95bcd4cdcb4cc5966050ffe2cffdff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } } peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } }
I20240502 01:52:45.173848 11339 leader_election.cc:288] T 1da6b311a749427f9fdd394bec7a033f P fd95bcd4cdcb4cc5966050ffe2cffdff [CANDIDATE]: Term 1 election: Requested vote from peers 34da849acb2045a9a2ef03933592dd08 (127.9.176.196:44521), 7b7a47245f294ce9a264fcc8f6351c0f (127.9.176.194:43645)
I20240502 01:52:45.173774 10511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1da6b311a749427f9fdd394bec7a033f" candidate_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34da849acb2045a9a2ef03933592dd08"
I20240502 01:52:45.174043 10511 raft_consensus.cc:3037] T 1da6b311a749427f9fdd394bec7a033f P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:45.174553 10245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1da6b311a749427f9fdd394bec7a033f" candidate_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b7a47245f294ce9a264fcc8f6351c0f"
I20240502 01:52:45.174813 10245 raft_consensus.cc:3037] T 1da6b311a749427f9fdd394bec7a033f P 7b7a47245f294ce9a264fcc8f6351c0f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:45.174968 10512 raft_consensus.cc:2454] T f72ee833c1d54106802dd9191e142284 P 34da849acb2045a9a2ef03933592dd08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd95bcd4cdcb4cc5966050ffe2cffdff in term 1.
I20240502 01:52:45.175709 10246 raft_consensus.cc:2454] T f72ee833c1d54106802dd9191e142284 P 7b7a47245f294ce9a264fcc8f6351c0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd95bcd4cdcb4cc5966050ffe2cffdff in term 1.
I20240502 01:52:45.175709 10711 leader_election.cc:302] T f72ee833c1d54106802dd9191e142284 P fd95bcd4cdcb4cc5966050ffe2cffdff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34da849acb2045a9a2ef03933592dd08, fd95bcd4cdcb4cc5966050ffe2cffdff; no voters: 
I20240502 01:52:45.176290 11339 raft_consensus.cc:2781] T f72ee833c1d54106802dd9191e142284 P fd95bcd4cdcb4cc5966050ffe2cffdff [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:45.176447 11339 raft_consensus.cc:686] T f72ee833c1d54106802dd9191e142284 P fd95bcd4cdcb4cc5966050ffe2cffdff [term 1 LEADER]: Becoming Leader. State: Replica: fd95bcd4cdcb4cc5966050ffe2cffdff, State: Running, Role: LEADER
I20240502 01:52:45.176755 11339 consensus_queue.cc:237] T f72ee833c1d54106802dd9191e142284 P fd95bcd4cdcb4cc5966050ffe2cffdff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } }
I20240502 01:52:45.176985 10511 raft_consensus.cc:2454] T 1da6b311a749427f9fdd394bec7a033f P 34da849acb2045a9a2ef03933592dd08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd95bcd4cdcb4cc5966050ffe2cffdff in term 1.
I20240502 01:52:45.177431 10711 leader_election.cc:302] T 1da6b311a749427f9fdd394bec7a033f P fd95bcd4cdcb4cc5966050ffe2cffdff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34da849acb2045a9a2ef03933592dd08, fd95bcd4cdcb4cc5966050ffe2cffdff; no voters: 
I20240502 01:52:45.177623 10245 raft_consensus.cc:2454] T 1da6b311a749427f9fdd394bec7a033f P 7b7a47245f294ce9a264fcc8f6351c0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd95bcd4cdcb4cc5966050ffe2cffdff in term 1.
I20240502 01:52:45.177774 11332 raft_consensus.cc:2781] T 1da6b311a749427f9fdd394bec7a033f P fd95bcd4cdcb4cc5966050ffe2cffdff [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:45.177932 11332 raft_consensus.cc:686] T 1da6b311a749427f9fdd394bec7a033f P fd95bcd4cdcb4cc5966050ffe2cffdff [term 1 LEADER]: Becoming Leader. State: Replica: fd95bcd4cdcb4cc5966050ffe2cffdff, State: Running, Role: LEADER
I20240502 01:52:45.178267 11332 consensus_queue.cc:237] T 1da6b311a749427f9fdd394bec7a033f P fd95bcd4cdcb4cc5966050ffe2cffdff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } } peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } }
I20240502 01:52:45.180640  9962 catalog_manager.cc:5484] T f72ee833c1d54106802dd9191e142284 P fd95bcd4cdcb4cc5966050ffe2cffdff reported cstate change: term changed from 0 to 1, leader changed from <none> to fd95bcd4cdcb4cc5966050ffe2cffdff (127.9.176.198). New cstate: current_term: 1 leader_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } health_report { overall_health: UNKNOWN } } }
I20240502 01:52:45.185909  9962 catalog_manager.cc:5484] T 1da6b311a749427f9fdd394bec7a033f P fd95bcd4cdcb4cc5966050ffe2cffdff reported cstate change: term changed from 0 to 1, leader changed from <none> to fd95bcd4cdcb4cc5966050ffe2cffdff (127.9.176.198). New cstate: current_term: 1 leader_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } health_report { overall_health: HEALTHY } } }
I20240502 01:52:45.253818 11359 raft_consensus.cc:482] T 968bb8eb10d44a29b33299898ebe453c P ede1dd598e3541138a19a3ea5c052bc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:45.254061 11359 raft_consensus.cc:504] T 968bb8eb10d44a29b33299898ebe453c P ede1dd598e3541138a19a3ea5c052bc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } }
I20240502 01:52:45.254928 11359 leader_election.cc:288] T 968bb8eb10d44a29b33299898ebe453c P ede1dd598e3541138a19a3ea5c052bc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a885359a82bd4515913f7b273591f851 (127.9.176.193:38373), a8c4a518929d4bb3a88bdfb1649493bd (127.9.176.197:44061)
I20240502 01:52:45.255649 10111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "968bb8eb10d44a29b33299898ebe453c" candidate_uuid: "ede1dd598e3541138a19a3ea5c052bc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a885359a82bd4515913f7b273591f851" is_pre_election: true
I20240502 01:52:45.255784 10643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "968bb8eb10d44a29b33299898ebe453c" candidate_uuid: "ede1dd598e3541138a19a3ea5c052bc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" is_pre_election: true
I20240502 01:52:45.255913 10111 raft_consensus.cc:2454] T 968bb8eb10d44a29b33299898ebe453c P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ede1dd598e3541138a19a3ea5c052bc9 in term 0.
I20240502 01:52:45.256091 10643 raft_consensus.cc:2454] T 968bb8eb10d44a29b33299898ebe453c P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ede1dd598e3541138a19a3ea5c052bc9 in term 0.
I20240502 01:52:45.256501 10313 leader_election.cc:302] T 968bb8eb10d44a29b33299898ebe453c P ede1dd598e3541138a19a3ea5c052bc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a885359a82bd4515913f7b273591f851, ede1dd598e3541138a19a3ea5c052bc9; no voters: 
I20240502 01:52:45.256850 11359 raft_consensus.cc:2781] T 968bb8eb10d44a29b33299898ebe453c P ede1dd598e3541138a19a3ea5c052bc9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:45.256963 11359 raft_consensus.cc:482] T 968bb8eb10d44a29b33299898ebe453c P ede1dd598e3541138a19a3ea5c052bc9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:45.257071 11359 raft_consensus.cc:3037] T 968bb8eb10d44a29b33299898ebe453c P ede1dd598e3541138a19a3ea5c052bc9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:45.259240 11359 raft_consensus.cc:504] T 968bb8eb10d44a29b33299898ebe453c P ede1dd598e3541138a19a3ea5c052bc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } }
I20240502 01:52:45.259959 11359 leader_election.cc:288] T 968bb8eb10d44a29b33299898ebe453c P ede1dd598e3541138a19a3ea5c052bc9 [CANDIDATE]: Term 1 election: Requested vote from peers a885359a82bd4515913f7b273591f851 (127.9.176.193:38373), a8c4a518929d4bb3a88bdfb1649493bd (127.9.176.197:44061)
I20240502 01:52:45.260694 10643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "968bb8eb10d44a29b33299898ebe453c" candidate_uuid: "ede1dd598e3541138a19a3ea5c052bc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8c4a518929d4bb3a88bdfb1649493bd"
I20240502 01:52:45.260671 10111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "968bb8eb10d44a29b33299898ebe453c" candidate_uuid: "ede1dd598e3541138a19a3ea5c052bc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a885359a82bd4515913f7b273591f851"
I20240502 01:52:45.260953 10111 raft_consensus.cc:3037] T 968bb8eb10d44a29b33299898ebe453c P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:45.260954 10643 raft_consensus.cc:3037] T 968bb8eb10d44a29b33299898ebe453c P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:45.264010 10111 raft_consensus.cc:2454] T 968bb8eb10d44a29b33299898ebe453c P a885359a82bd4515913f7b273591f851 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ede1dd598e3541138a19a3ea5c052bc9 in term 1.
I20240502 01:52:45.264248 10643 raft_consensus.cc:2454] T 968bb8eb10d44a29b33299898ebe453c P a8c4a518929d4bb3a88bdfb1649493bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ede1dd598e3541138a19a3ea5c052bc9 in term 1.
I20240502 01:52:45.264640 10313 leader_election.cc:302] T 968bb8eb10d44a29b33299898ebe453c P ede1dd598e3541138a19a3ea5c052bc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a885359a82bd4515913f7b273591f851, ede1dd598e3541138a19a3ea5c052bc9; no voters: 
I20240502 01:52:45.265004 11359 raft_consensus.cc:2781] T 968bb8eb10d44a29b33299898ebe453c P ede1dd598e3541138a19a3ea5c052bc9 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:45.265589 11359 raft_consensus.cc:686] T 968bb8eb10d44a29b33299898ebe453c P ede1dd598e3541138a19a3ea5c052bc9 [term 1 LEADER]: Becoming Leader. State: Replica: ede1dd598e3541138a19a3ea5c052bc9, State: Running, Role: LEADER
I20240502 01:52:45.266117 11359 consensus_queue.cc:237] T 968bb8eb10d44a29b33299898ebe453c P ede1dd598e3541138a19a3ea5c052bc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } }
I20240502 01:52:45.270232  9962 catalog_manager.cc:5484] T 968bb8eb10d44a29b33299898ebe453c P ede1dd598e3541138a19a3ea5c052bc9 reported cstate change: term changed from 0 to 1, leader changed from <none> to ede1dd598e3541138a19a3ea5c052bc9 (127.9.176.195). New cstate: current_term: 1 leader_uuid: "ede1dd598e3541138a19a3ea5c052bc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } health_report { overall_health: HEALTHY } } }
I20240502 01:52:45.589437 11332 consensus_queue.cc:1035] T 1da6b311a749427f9fdd394bec7a033f P fd95bcd4cdcb4cc5966050ffe2cffdff [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:45.608597 11188 consensus_queue.cc:1035] T 1da6b311a749427f9fdd394bec7a033f P fd95bcd4cdcb4cc5966050ffe2cffdff [LEADER]: Connected to new peer: Peer: permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:45.629928 11339 consensus_queue.cc:1035] T f72ee833c1d54106802dd9191e142284 P fd95bcd4cdcb4cc5966050ffe2cffdff [LEADER]: Connected to new peer: Peer: permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:45.636134 11332 consensus_queue.cc:1035] T f72ee833c1d54106802dd9191e142284 P fd95bcd4cdcb4cc5966050ffe2cffdff [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:45.665513 11359 consensus_queue.cc:1035] T 968bb8eb10d44a29b33299898ebe453c P ede1dd598e3541138a19a3ea5c052bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:45.671931 11359 consensus_queue.cc:1035] T 968bb8eb10d44a29b33299898ebe453c P ede1dd598e3541138a19a3ea5c052bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:52:45.717525 11367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:45.717962 11367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:45.726322 11367 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'

INSERT report
    rows total: 2000
    time total: 224.593 ms
  time per row: 0.112296 ms
I20240502 01:52:46.136725  9961 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56524:
name: "rebalance_test_table_6"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 01:52:46.147984 10358 tablet_service.cc:1467] Processing CreateTablet for tablet 6488dcdd5ab04e35bbd720f4985ceb41 (DEFAULT_TABLE table=rebalance_test_table_6 [id=3c954a8d05e34cb4bb59db555f4fc6ae]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:46.148298 10492 tablet_service.cc:1467] Processing CreateTablet for tablet 09a892cf118d41d4bfffc1576731d54c (DEFAULT_TABLE table=rebalance_test_table_6 [id=3c954a8d05e34cb4bb59db555f4fc6ae]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:46.148384 10092 tablet_service.cc:1467] Processing CreateTablet for tablet 6488dcdd5ab04e35bbd720f4985ceb41 (DEFAULT_TABLE table=rebalance_test_table_6 [id=3c954a8d05e34cb4bb59db555f4fc6ae]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:46.148609 10358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6488dcdd5ab04e35bbd720f4985ceb41. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:46.148741 10492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09a892cf118d41d4bfffc1576731d54c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:46.148876 10092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6488dcdd5ab04e35bbd720f4985ceb41. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:46.155443 10759 tablet_service.cc:1467] Processing CreateTablet for tablet 09a892cf118d41d4bfffc1576731d54c (DEFAULT_TABLE table=rebalance_test_table_6 [id=3c954a8d05e34cb4bb59db555f4fc6ae]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:46.155723 10624 tablet_service.cc:1467] Processing CreateTablet for tablet 6488dcdd5ab04e35bbd720f4985ceb41 (DEFAULT_TABLE table=rebalance_test_table_6 [id=3c954a8d05e34cb4bb59db555f4fc6ae]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:46.155915 10225 tablet_service.cc:1467] Processing CreateTablet for tablet 09a892cf118d41d4bfffc1576731d54c (DEFAULT_TABLE table=rebalance_test_table_6 [id=3c954a8d05e34cb4bb59db555f4fc6ae]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:46.156051 10759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09a892cf118d41d4bfffc1576731d54c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:46.156183 10624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6488dcdd5ab04e35bbd720f4985ceb41. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:46.156338 10225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09a892cf118d41d4bfffc1576731d54c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:46.160281 10623 tablet_service.cc:1467] Processing CreateTablet for tablet de6e1081d71641248b0657bbdb824af8 (DEFAULT_TABLE table=rebalance_test_table_6 [id=3c954a8d05e34cb4bb59db555f4fc6ae]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:46.160712 10623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de6e1081d71641248b0657bbdb824af8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:46.163645 11432 tablet_bootstrap.cc:492] T 6488dcdd5ab04e35bbd720f4985ceb41 P a8c4a518929d4bb3a88bdfb1649493bd: Bootstrap starting.
I20240502 01:52:46.163647 10091 tablet_service.cc:1467] Processing CreateTablet for tablet de6e1081d71641248b0657bbdb824af8 (DEFAULT_TABLE table=rebalance_test_table_6 [id=3c954a8d05e34cb4bb59db555f4fc6ae]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:46.164206 10091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de6e1081d71641248b0657bbdb824af8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:46.165241 11433 tablet_bootstrap.cc:492] T 09a892cf118d41d4bfffc1576731d54c P 7b7a47245f294ce9a264fcc8f6351c0f: Bootstrap starting.
I20240502 01:52:46.165486 10357 tablet_service.cc:1467] Processing CreateTablet for tablet de6e1081d71641248b0657bbdb824af8 (DEFAULT_TABLE table=rebalance_test_table_6 [id=3c954a8d05e34cb4bb59db555f4fc6ae]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:46.166031 10357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de6e1081d71641248b0657bbdb824af8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:46.166733 11432 tablet_bootstrap.cc:654] T 6488dcdd5ab04e35bbd720f4985ceb41 P a8c4a518929d4bb3a88bdfb1649493bd: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:46.169493 11433 tablet_bootstrap.cc:654] T 09a892cf118d41d4bfffc1576731d54c P 7b7a47245f294ce9a264fcc8f6351c0f: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:46.171346 11434 tablet_bootstrap.cc:492] T 6488dcdd5ab04e35bbd720f4985ceb41 P ede1dd598e3541138a19a3ea5c052bc9: Bootstrap starting.
I20240502 01:52:46.174084 11434 tablet_bootstrap.cc:654] T 6488dcdd5ab04e35bbd720f4985ceb41 P ede1dd598e3541138a19a3ea5c052bc9: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:46.176530 11433 tablet_bootstrap.cc:492] T 09a892cf118d41d4bfffc1576731d54c P 7b7a47245f294ce9a264fcc8f6351c0f: No bootstrap required, opened a new log
I20240502 01:52:46.176723 11433 ts_tablet_manager.cc:1367] T 09a892cf118d41d4bfffc1576731d54c P 7b7a47245f294ce9a264fcc8f6351c0f: Time spent bootstrapping tablet: real 0.012s	user 0.005s	sys 0.000s
I20240502 01:52:46.178107 11433 raft_consensus.cc:348] T 09a892cf118d41d4bfffc1576731d54c P 7b7a47245f294ce9a264fcc8f6351c0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } } peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } }
I20240502 01:52:46.179028 11433 raft_consensus.cc:374] T 09a892cf118d41d4bfffc1576731d54c P 7b7a47245f294ce9a264fcc8f6351c0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:46.179339 11433 raft_consensus.cc:729] T 09a892cf118d41d4bfffc1576731d54c P 7b7a47245f294ce9a264fcc8f6351c0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b7a47245f294ce9a264fcc8f6351c0f, State: Initialized, Role: FOLLOWER
I20240502 01:52:46.179891 11433 consensus_queue.cc:260] T 09a892cf118d41d4bfffc1576731d54c P 7b7a47245f294ce9a264fcc8f6351c0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } } peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } }
I20240502 01:52:46.180567 11437 tablet_bootstrap.cc:492] T 6488dcdd5ab04e35bbd720f4985ceb41 P a885359a82bd4515913f7b273591f851: Bootstrap starting.
I20240502 01:52:46.181895 11433 ts_tablet_manager.cc:1398] T 09a892cf118d41d4bfffc1576731d54c P 7b7a47245f294ce9a264fcc8f6351c0f: Time spent starting tablet: real 0.005s	user 0.003s	sys 0.000s
I20240502 01:52:46.183384 11437 tablet_bootstrap.cc:654] T 6488dcdd5ab04e35bbd720f4985ceb41 P a885359a82bd4515913f7b273591f851: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:46.185667 11432 tablet_bootstrap.cc:492] T 6488dcdd5ab04e35bbd720f4985ceb41 P a8c4a518929d4bb3a88bdfb1649493bd: No bootstrap required, opened a new log
I20240502 01:52:46.185895 11432 ts_tablet_manager.cc:1367] T 6488dcdd5ab04e35bbd720f4985ceb41 P a8c4a518929d4bb3a88bdfb1649493bd: Time spent bootstrapping tablet: real 0.022s	user 0.004s	sys 0.005s
I20240502 01:52:46.187047 11432 raft_consensus.cc:348] T 6488dcdd5ab04e35bbd720f4985ceb41 P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } }
I20240502 01:52:46.187350 11432 raft_consensus.cc:374] T 6488dcdd5ab04e35bbd720f4985ceb41 P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:46.187474 11432 raft_consensus.cc:729] T 6488dcdd5ab04e35bbd720f4985ceb41 P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8c4a518929d4bb3a88bdfb1649493bd, State: Initialized, Role: FOLLOWER
I20240502 01:52:46.187510 11430 tablet_bootstrap.cc:492] T 09a892cf118d41d4bfffc1576731d54c P 34da849acb2045a9a2ef03933592dd08: Bootstrap starting.
I20240502 01:52:46.187824 11432 consensus_queue.cc:260] T 6488dcdd5ab04e35bbd720f4985ceb41 P a8c4a518929d4bb3a88bdfb1649493bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } }
I20240502 01:52:46.189780 11432 ts_tablet_manager.cc:1398] T 6488dcdd5ab04e35bbd720f4985ceb41 P a8c4a518929d4bb3a88bdfb1649493bd: Time spent starting tablet: real 0.004s	user 0.003s	sys 0.001s
I20240502 01:52:46.190212 11432 tablet_bootstrap.cc:492] T de6e1081d71641248b0657bbdb824af8 P a8c4a518929d4bb3a88bdfb1649493bd: Bootstrap starting.
I20240502 01:52:46.191289 11430 tablet_bootstrap.cc:654] T 09a892cf118d41d4bfffc1576731d54c P 34da849acb2045a9a2ef03933592dd08: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:46.193217 11432 tablet_bootstrap.cc:654] T de6e1081d71641248b0657bbdb824af8 P a8c4a518929d4bb3a88bdfb1649493bd: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:46.198305 11439 tablet_bootstrap.cc:492] T 09a892cf118d41d4bfffc1576731d54c P fd95bcd4cdcb4cc5966050ffe2cffdff: Bootstrap starting.
I20240502 01:52:46.200409 11434 tablet_bootstrap.cc:492] T 6488dcdd5ab04e35bbd720f4985ceb41 P ede1dd598e3541138a19a3ea5c052bc9: No bootstrap required, opened a new log
I20240502 01:52:46.200610 11434 ts_tablet_manager.cc:1367] T 6488dcdd5ab04e35bbd720f4985ceb41 P ede1dd598e3541138a19a3ea5c052bc9: Time spent bootstrapping tablet: real 0.029s	user 0.010s	sys 0.015s
I20240502 01:52:46.201625 11434 raft_consensus.cc:348] T 6488dcdd5ab04e35bbd720f4985ceb41 P ede1dd598e3541138a19a3ea5c052bc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } }
I20240502 01:52:46.201855 11434 raft_consensus.cc:374] T 6488dcdd5ab04e35bbd720f4985ceb41 P ede1dd598e3541138a19a3ea5c052bc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:46.201964 11434 raft_consensus.cc:729] T 6488dcdd5ab04e35bbd720f4985ceb41 P ede1dd598e3541138a19a3ea5c052bc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ede1dd598e3541138a19a3ea5c052bc9, State: Initialized, Role: FOLLOWER
I20240502 01:52:46.202052 11432 tablet_bootstrap.cc:492] T de6e1081d71641248b0657bbdb824af8 P a8c4a518929d4bb3a88bdfb1649493bd: No bootstrap required, opened a new log
I20240502 01:52:46.202220 11432 ts_tablet_manager.cc:1367] T de6e1081d71641248b0657bbdb824af8 P a8c4a518929d4bb3a88bdfb1649493bd: Time spent bootstrapping tablet: real 0.012s	user 0.004s	sys 0.001s
I20240502 01:52:46.202301 11434 consensus_queue.cc:260] T 6488dcdd5ab04e35bbd720f4985ceb41 P ede1dd598e3541138a19a3ea5c052bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } }
I20240502 01:52:46.202489 11439 tablet_bootstrap.cc:654] T 09a892cf118d41d4bfffc1576731d54c P fd95bcd4cdcb4cc5966050ffe2cffdff: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:46.202714 11430 tablet_bootstrap.cc:492] T 09a892cf118d41d4bfffc1576731d54c P 34da849acb2045a9a2ef03933592dd08: No bootstrap required, opened a new log
I20240502 01:52:46.202908 11430 ts_tablet_manager.cc:1367] T 09a892cf118d41d4bfffc1576731d54c P 34da849acb2045a9a2ef03933592dd08: Time spent bootstrapping tablet: real 0.016s	user 0.007s	sys 0.000s
I20240502 01:52:46.203104 11434 ts_tablet_manager.cc:1398] T 6488dcdd5ab04e35bbd720f4985ceb41 P ede1dd598e3541138a19a3ea5c052bc9: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.000s
I20240502 01:52:46.202998 11432 raft_consensus.cc:348] T de6e1081d71641248b0657bbdb824af8 P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } }
I20240502 01:52:46.203225 11432 raft_consensus.cc:374] T de6e1081d71641248b0657bbdb824af8 P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:46.203418 11432 raft_consensus.cc:729] T de6e1081d71641248b0657bbdb824af8 P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8c4a518929d4bb3a88bdfb1649493bd, State: Initialized, Role: FOLLOWER
I20240502 01:52:46.203465 11434 tablet_bootstrap.cc:492] T de6e1081d71641248b0657bbdb824af8 P ede1dd598e3541138a19a3ea5c052bc9: Bootstrap starting.
I20240502 01:52:46.203717 11432 consensus_queue.cc:260] T de6e1081d71641248b0657bbdb824af8 P a8c4a518929d4bb3a88bdfb1649493bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } }
I20240502 01:52:46.204313 11432 ts_tablet_manager.cc:1398] T de6e1081d71641248b0657bbdb824af8 P a8c4a518929d4bb3a88bdfb1649493bd: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 01:52:46.205090 11434 tablet_bootstrap.cc:654] T de6e1081d71641248b0657bbdb824af8 P ede1dd598e3541138a19a3ea5c052bc9: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:46.203866 11430 raft_consensus.cc:348] T 09a892cf118d41d4bfffc1576731d54c P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } } peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } }
I20240502 01:52:46.205731 11430 raft_consensus.cc:374] T 09a892cf118d41d4bfffc1576731d54c P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:46.205830 11430 raft_consensus.cc:729] T 09a892cf118d41d4bfffc1576731d54c P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34da849acb2045a9a2ef03933592dd08, State: Initialized, Role: FOLLOWER
I20240502 01:52:46.206091 11430 consensus_queue.cc:260] T 09a892cf118d41d4bfffc1576731d54c P 34da849acb2045a9a2ef03933592dd08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } } peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } }
I20240502 01:52:46.206876 11430 ts_tablet_manager.cc:1398] T 09a892cf118d41d4bfffc1576731d54c P 34da849acb2045a9a2ef03933592dd08: Time spent starting tablet: real 0.004s	user 0.001s	sys 0.000s
I20240502 01:52:46.207501 11437 tablet_bootstrap.cc:492] T 6488dcdd5ab04e35bbd720f4985ceb41 P a885359a82bd4515913f7b273591f851: No bootstrap required, opened a new log
I20240502 01:52:46.209173 11437 ts_tablet_manager.cc:1367] T 6488dcdd5ab04e35bbd720f4985ceb41 P a885359a82bd4515913f7b273591f851: Time spent bootstrapping tablet: real 0.029s	user 0.000s	sys 0.010s
I20240502 01:52:46.210053 11437 raft_consensus.cc:348] T 6488dcdd5ab04e35bbd720f4985ceb41 P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } }
I20240502 01:52:46.210338 11437 raft_consensus.cc:374] T 6488dcdd5ab04e35bbd720f4985ceb41 P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:46.210477 11437 raft_consensus.cc:729] T 6488dcdd5ab04e35bbd720f4985ceb41 P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a885359a82bd4515913f7b273591f851, State: Initialized, Role: FOLLOWER
I20240502 01:52:46.210788 11437 consensus_queue.cc:260] T 6488dcdd5ab04e35bbd720f4985ceb41 P a885359a82bd4515913f7b273591f851 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } }
I20240502 01:52:46.211002 11434 tablet_bootstrap.cc:492] T de6e1081d71641248b0657bbdb824af8 P ede1dd598e3541138a19a3ea5c052bc9: No bootstrap required, opened a new log
I20240502 01:52:46.211607 11437 ts_tablet_manager.cc:1398] T 6488dcdd5ab04e35bbd720f4985ceb41 P a885359a82bd4515913f7b273591f851: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240502 01:52:46.211187 11434 ts_tablet_manager.cc:1367] T de6e1081d71641248b0657bbdb824af8 P ede1dd598e3541138a19a3ea5c052bc9: Time spent bootstrapping tablet: real 0.008s	user 0.005s	sys 0.000s
I20240502 01:52:46.212340 11437 tablet_bootstrap.cc:492] T de6e1081d71641248b0657bbdb824af8 P a885359a82bd4515913f7b273591f851: Bootstrap starting.
I20240502 01:52:46.212860 11434 raft_consensus.cc:348] T de6e1081d71641248b0657bbdb824af8 P ede1dd598e3541138a19a3ea5c052bc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } }
I20240502 01:52:46.213095 11434 raft_consensus.cc:374] T de6e1081d71641248b0657bbdb824af8 P ede1dd598e3541138a19a3ea5c052bc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:46.213187 11434 raft_consensus.cc:729] T de6e1081d71641248b0657bbdb824af8 P ede1dd598e3541138a19a3ea5c052bc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ede1dd598e3541138a19a3ea5c052bc9, State: Initialized, Role: FOLLOWER
I20240502 01:52:46.213451 11434 consensus_queue.cc:260] T de6e1081d71641248b0657bbdb824af8 P ede1dd598e3541138a19a3ea5c052bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } }
I20240502 01:52:46.214093 11434 ts_tablet_manager.cc:1398] T de6e1081d71641248b0657bbdb824af8 P ede1dd598e3541138a19a3ea5c052bc9: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 01:52:46.214401 11437 tablet_bootstrap.cc:654] T de6e1081d71641248b0657bbdb824af8 P a885359a82bd4515913f7b273591f851: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:46.220957 11439 tablet_bootstrap.cc:492] T 09a892cf118d41d4bfffc1576731d54c P fd95bcd4cdcb4cc5966050ffe2cffdff: No bootstrap required, opened a new log
I20240502 01:52:46.221132 11439 ts_tablet_manager.cc:1367] T 09a892cf118d41d4bfffc1576731d54c P fd95bcd4cdcb4cc5966050ffe2cffdff: Time spent bootstrapping tablet: real 0.023s	user 0.006s	sys 0.006s
I20240502 01:52:46.221288 11437 tablet_bootstrap.cc:492] T de6e1081d71641248b0657bbdb824af8 P a885359a82bd4515913f7b273591f851: No bootstrap required, opened a new log
I20240502 01:52:46.221457 11437 ts_tablet_manager.cc:1367] T de6e1081d71641248b0657bbdb824af8 P a885359a82bd4515913f7b273591f851: Time spent bootstrapping tablet: real 0.010s	user 0.000s	sys 0.007s
I20240502 01:52:46.221859 11439 raft_consensus.cc:348] T 09a892cf118d41d4bfffc1576731d54c P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } } peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } }
I20240502 01:52:46.222040 11439 raft_consensus.cc:374] T 09a892cf118d41d4bfffc1576731d54c P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:46.222136 11439 raft_consensus.cc:729] T 09a892cf118d41d4bfffc1576731d54c P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd95bcd4cdcb4cc5966050ffe2cffdff, State: Initialized, Role: FOLLOWER
I20240502 01:52:46.222112 11437 raft_consensus.cc:348] T de6e1081d71641248b0657bbdb824af8 P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } }
I20240502 01:52:46.222383 11437 raft_consensus.cc:374] T de6e1081d71641248b0657bbdb824af8 P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:46.222460 11437 raft_consensus.cc:729] T de6e1081d71641248b0657bbdb824af8 P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a885359a82bd4515913f7b273591f851, State: Initialized, Role: FOLLOWER
I20240502 01:52:46.222391 11439 consensus_queue.cc:260] T 09a892cf118d41d4bfffc1576731d54c P fd95bcd4cdcb4cc5966050ffe2cffdff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } } peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } }
I20240502 01:52:46.222729 11437 consensus_queue.cc:260] T de6e1081d71641248b0657bbdb824af8 P a885359a82bd4515913f7b273591f851 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } }
I20240502 01:52:46.222963 11439 ts_tablet_manager.cc:1398] T 09a892cf118d41d4bfffc1576731d54c P fd95bcd4cdcb4cc5966050ffe2cffdff: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 01:52:46.223341 11437 ts_tablet_manager.cc:1398] T de6e1081d71641248b0657bbdb824af8 P a885359a82bd4515913f7b273591f851: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240502 01:52:46.289237 10942 raft_consensus.cc:482] T de6e1081d71641248b0657bbdb824af8 P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:46.289520 10942 raft_consensus.cc:504] T de6e1081d71641248b0657bbdb824af8 P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } }
I20240502 01:52:46.290387 10942 leader_election.cc:288] T de6e1081d71641248b0657bbdb824af8 P a885359a82bd4515913f7b273591f851 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a8c4a518929d4bb3a88bdfb1649493bd (127.9.176.197:44061), ede1dd598e3541138a19a3ea5c052bc9 (127.9.176.195:38701)
I20240502 01:52:46.290813 10643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de6e1081d71641248b0657bbdb824af8" candidate_uuid: "a885359a82bd4515913f7b273591f851" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" is_pre_election: true
I20240502 01:52:46.290937 10378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de6e1081d71641248b0657bbdb824af8" candidate_uuid: "a885359a82bd4515913f7b273591f851" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ede1dd598e3541138a19a3ea5c052bc9" is_pre_election: true
I20240502 01:52:46.291093 10643 raft_consensus.cc:2454] T de6e1081d71641248b0657bbdb824af8 P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a885359a82bd4515913f7b273591f851 in term 0.
I20240502 01:52:46.291172 10378 raft_consensus.cc:2454] T de6e1081d71641248b0657bbdb824af8 P ede1dd598e3541138a19a3ea5c052bc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a885359a82bd4515913f7b273591f851 in term 0.
I20240502 01:52:46.291599 10046 leader_election.cc:302] T de6e1081d71641248b0657bbdb824af8 P a885359a82bd4515913f7b273591f851 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a885359a82bd4515913f7b273591f851, a8c4a518929d4bb3a88bdfb1649493bd; no voters: 
I20240502 01:52:46.291905 10942 raft_consensus.cc:2781] T de6e1081d71641248b0657bbdb824af8 P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:46.291999 10942 raft_consensus.cc:482] T de6e1081d71641248b0657bbdb824af8 P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:46.292130 10942 raft_consensus.cc:3037] T de6e1081d71641248b0657bbdb824af8 P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:46.294107 10942 raft_consensus.cc:504] T de6e1081d71641248b0657bbdb824af8 P a885359a82bd4515913f7b273591f851 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } }
I20240502 01:52:46.294785 10942 leader_election.cc:288] T de6e1081d71641248b0657bbdb824af8 P a885359a82bd4515913f7b273591f851 [CANDIDATE]: Term 1 election: Requested vote from peers a8c4a518929d4bb3a88bdfb1649493bd (127.9.176.197:44061), ede1dd598e3541138a19a3ea5c052bc9 (127.9.176.195:38701)
I20240502 01:52:46.295257 10643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de6e1081d71641248b0657bbdb824af8" candidate_uuid: "a885359a82bd4515913f7b273591f851" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8c4a518929d4bb3a88bdfb1649493bd"
I20240502 01:52:46.295331 10378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de6e1081d71641248b0657bbdb824af8" candidate_uuid: "a885359a82bd4515913f7b273591f851" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ede1dd598e3541138a19a3ea5c052bc9"
I20240502 01:52:46.295446 10643 raft_consensus.cc:3037] T de6e1081d71641248b0657bbdb824af8 P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:46.295589 10378 raft_consensus.cc:3037] T de6e1081d71641248b0657bbdb824af8 P ede1dd598e3541138a19a3ea5c052bc9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:46.297631 10378 raft_consensus.cc:2454] T de6e1081d71641248b0657bbdb824af8 P ede1dd598e3541138a19a3ea5c052bc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a885359a82bd4515913f7b273591f851 in term 1.
I20240502 01:52:46.297719 10643 raft_consensus.cc:2454] T de6e1081d71641248b0657bbdb824af8 P a8c4a518929d4bb3a88bdfb1649493bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a885359a82bd4515913f7b273591f851 in term 1.
I20240502 01:52:46.298348 10046 leader_election.cc:302] T de6e1081d71641248b0657bbdb824af8 P a885359a82bd4515913f7b273591f851 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a885359a82bd4515913f7b273591f851, a8c4a518929d4bb3a88bdfb1649493bd; no voters: 
I20240502 01:52:46.298683 10942 raft_consensus.cc:2781] T de6e1081d71641248b0657bbdb824af8 P a885359a82bd4515913f7b273591f851 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:46.299118 10942 raft_consensus.cc:686] T de6e1081d71641248b0657bbdb824af8 P a885359a82bd4515913f7b273591f851 [term 1 LEADER]: Becoming Leader. State: Replica: a885359a82bd4515913f7b273591f851, State: Running, Role: LEADER
I20240502 01:52:46.299398 10942 consensus_queue.cc:237] T de6e1081d71641248b0657bbdb824af8 P a885359a82bd4515913f7b273591f851 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } }
I20240502 01:52:46.302901  9958 catalog_manager.cc:5484] T de6e1081d71641248b0657bbdb824af8 P a885359a82bd4515913f7b273591f851 reported cstate change: term changed from 0 to 1, leader changed from <none> to a885359a82bd4515913f7b273591f851 (127.9.176.193). New cstate: current_term: 1 leader_uuid: "a885359a82bd4515913f7b273591f851" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } health_report { overall_health: UNKNOWN } } }
I20240502 01:52:46.317823 11339 raft_consensus.cc:482] T 09a892cf118d41d4bfffc1576731d54c P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:46.318023 11339 raft_consensus.cc:504] T 09a892cf118d41d4bfffc1576731d54c P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } } peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } }
I20240502 01:52:46.318795 11339 leader_election.cc:288] T 09a892cf118d41d4bfffc1576731d54c P fd95bcd4cdcb4cc5966050ffe2cffdff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 34da849acb2045a9a2ef03933592dd08 (127.9.176.196:44521), 7b7a47245f294ce9a264fcc8f6351c0f (127.9.176.194:43645)
I20240502 01:52:46.319362 10512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09a892cf118d41d4bfffc1576731d54c" candidate_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34da849acb2045a9a2ef03933592dd08" is_pre_election: true
I20240502 01:52:46.319490 10245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09a892cf118d41d4bfffc1576731d54c" candidate_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" is_pre_election: true
I20240502 01:52:46.319659 10512 raft_consensus.cc:2454] T 09a892cf118d41d4bfffc1576731d54c P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd95bcd4cdcb4cc5966050ffe2cffdff in term 0.
I20240502 01:52:46.319773 10245 raft_consensus.cc:2454] T 09a892cf118d41d4bfffc1576731d54c P 7b7a47245f294ce9a264fcc8f6351c0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd95bcd4cdcb4cc5966050ffe2cffdff in term 0.
I20240502 01:52:46.320150 10711 leader_election.cc:302] T 09a892cf118d41d4bfffc1576731d54c P fd95bcd4cdcb4cc5966050ffe2cffdff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34da849acb2045a9a2ef03933592dd08, fd95bcd4cdcb4cc5966050ffe2cffdff; no voters: 
I20240502 01:52:46.320427 11339 raft_consensus.cc:2781] T 09a892cf118d41d4bfffc1576731d54c P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:46.320557 11339 raft_consensus.cc:482] T 09a892cf118d41d4bfffc1576731d54c P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:46.320654 11339 raft_consensus.cc:3037] T 09a892cf118d41d4bfffc1576731d54c P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:46.322654 11339 raft_consensus.cc:504] T 09a892cf118d41d4bfffc1576731d54c P fd95bcd4cdcb4cc5966050ffe2cffdff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } } peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } }
I20240502 01:52:46.323302 11339 leader_election.cc:288] T 09a892cf118d41d4bfffc1576731d54c P fd95bcd4cdcb4cc5966050ffe2cffdff [CANDIDATE]: Term 1 election: Requested vote from peers 34da849acb2045a9a2ef03933592dd08 (127.9.176.196:44521), 7b7a47245f294ce9a264fcc8f6351c0f (127.9.176.194:43645)
I20240502 01:52:46.323848 10512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09a892cf118d41d4bfffc1576731d54c" candidate_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34da849acb2045a9a2ef03933592dd08"
I20240502 01:52:46.323954 10245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09a892cf118d41d4bfffc1576731d54c" candidate_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b7a47245f294ce9a264fcc8f6351c0f"
I20240502 01:52:46.324057 10512 raft_consensus.cc:3037] T 09a892cf118d41d4bfffc1576731d54c P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:46.324214 10245 raft_consensus.cc:3037] T 09a892cf118d41d4bfffc1576731d54c P 7b7a47245f294ce9a264fcc8f6351c0f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:46.326059 10512 raft_consensus.cc:2454] T 09a892cf118d41d4bfffc1576731d54c P 34da849acb2045a9a2ef03933592dd08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd95bcd4cdcb4cc5966050ffe2cffdff in term 1.
I20240502 01:52:46.326536 10711 leader_election.cc:302] T 09a892cf118d41d4bfffc1576731d54c P fd95bcd4cdcb4cc5966050ffe2cffdff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34da849acb2045a9a2ef03933592dd08, fd95bcd4cdcb4cc5966050ffe2cffdff; no voters: 
I20240502 01:52:46.326561 10245 raft_consensus.cc:2454] T 09a892cf118d41d4bfffc1576731d54c P 7b7a47245f294ce9a264fcc8f6351c0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd95bcd4cdcb4cc5966050ffe2cffdff in term 1.
I20240502 01:52:46.326855 11339 raft_consensus.cc:2781] T 09a892cf118d41d4bfffc1576731d54c P fd95bcd4cdcb4cc5966050ffe2cffdff [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:46.327005 11339 raft_consensus.cc:686] T 09a892cf118d41d4bfffc1576731d54c P fd95bcd4cdcb4cc5966050ffe2cffdff [term 1 LEADER]: Becoming Leader. State: Replica: fd95bcd4cdcb4cc5966050ffe2cffdff, State: Running, Role: LEADER
I20240502 01:52:46.327304 11339 consensus_queue.cc:237] T 09a892cf118d41d4bfffc1576731d54c P fd95bcd4cdcb4cc5966050ffe2cffdff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } } peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } }
I20240502 01:52:46.330565  9958 catalog_manager.cc:5484] T 09a892cf118d41d4bfffc1576731d54c P fd95bcd4cdcb4cc5966050ffe2cffdff reported cstate change: term changed from 0 to 1, leader changed from <none> to fd95bcd4cdcb4cc5966050ffe2cffdff (127.9.176.198). New cstate: current_term: 1 leader_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } health_report { overall_health: UNKNOWN } } }
I20240502 01:52:46.381453 10942 raft_consensus.cc:482] T 6488dcdd5ab04e35bbd720f4985ceb41 P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:46.381664 10942 raft_consensus.cc:504] T 6488dcdd5ab04e35bbd720f4985ceb41 P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } }
I20240502 01:52:46.382424 10942 leader_election.cc:288] T 6488dcdd5ab04e35bbd720f4985ceb41 P a885359a82bd4515913f7b273591f851 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ede1dd598e3541138a19a3ea5c052bc9 (127.9.176.195:38701), a8c4a518929d4bb3a88bdfb1649493bd (127.9.176.197:44061)
I20240502 01:52:46.383054 10643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6488dcdd5ab04e35bbd720f4985ceb41" candidate_uuid: "a885359a82bd4515913f7b273591f851" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" is_pre_election: true
I20240502 01:52:46.383111 10378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6488dcdd5ab04e35bbd720f4985ceb41" candidate_uuid: "a885359a82bd4515913f7b273591f851" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ede1dd598e3541138a19a3ea5c052bc9" is_pre_election: true
I20240502 01:52:46.383333 10643 raft_consensus.cc:2454] T 6488dcdd5ab04e35bbd720f4985ceb41 P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a885359a82bd4515913f7b273591f851 in term 0.
I20240502 01:52:46.383379 10378 raft_consensus.cc:2454] T 6488dcdd5ab04e35bbd720f4985ceb41 P ede1dd598e3541138a19a3ea5c052bc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a885359a82bd4515913f7b273591f851 in term 0.
I20240502 01:52:46.383826 10046 leader_election.cc:302] T 6488dcdd5ab04e35bbd720f4985ceb41 P a885359a82bd4515913f7b273591f851 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a885359a82bd4515913f7b273591f851, a8c4a518929d4bb3a88bdfb1649493bd; no voters: 
I20240502 01:52:46.384145 10942 raft_consensus.cc:2781] T 6488dcdd5ab04e35bbd720f4985ceb41 P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:46.384303 10942 raft_consensus.cc:482] T 6488dcdd5ab04e35bbd720f4985ceb41 P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:46.384403 10942 raft_consensus.cc:3037] T 6488dcdd5ab04e35bbd720f4985ceb41 P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:46.386576 10942 raft_consensus.cc:504] T 6488dcdd5ab04e35bbd720f4985ceb41 P a885359a82bd4515913f7b273591f851 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } }
I20240502 01:52:46.387485 10942 leader_election.cc:288] T 6488dcdd5ab04e35bbd720f4985ceb41 P a885359a82bd4515913f7b273591f851 [CANDIDATE]: Term 1 election: Requested vote from peers ede1dd598e3541138a19a3ea5c052bc9 (127.9.176.195:38701), a8c4a518929d4bb3a88bdfb1649493bd (127.9.176.197:44061)
I20240502 01:52:46.388069 10378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6488dcdd5ab04e35bbd720f4985ceb41" candidate_uuid: "a885359a82bd4515913f7b273591f851" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ede1dd598e3541138a19a3ea5c052bc9"
I20240502 01:52:46.388075 10643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6488dcdd5ab04e35bbd720f4985ceb41" candidate_uuid: "a885359a82bd4515913f7b273591f851" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8c4a518929d4bb3a88bdfb1649493bd"
I20240502 01:52:46.388299 10643 raft_consensus.cc:3037] T 6488dcdd5ab04e35bbd720f4985ceb41 P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:46.388288 10378 raft_consensus.cc:3037] T 6488dcdd5ab04e35bbd720f4985ceb41 P ede1dd598e3541138a19a3ea5c052bc9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:46.390419 10378 raft_consensus.cc:2454] T 6488dcdd5ab04e35bbd720f4985ceb41 P ede1dd598e3541138a19a3ea5c052bc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a885359a82bd4515913f7b273591f851 in term 1.
I20240502 01:52:46.390522 10643 raft_consensus.cc:2454] T 6488dcdd5ab04e35bbd720f4985ceb41 P a8c4a518929d4bb3a88bdfb1649493bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a885359a82bd4515913f7b273591f851 in term 1.
I20240502 01:52:46.391052 10046 leader_election.cc:302] T 6488dcdd5ab04e35bbd720f4985ceb41 P a885359a82bd4515913f7b273591f851 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a885359a82bd4515913f7b273591f851, a8c4a518929d4bb3a88bdfb1649493bd; no voters: 
I20240502 01:52:46.391456 10942 raft_consensus.cc:2781] T 6488dcdd5ab04e35bbd720f4985ceb41 P a885359a82bd4515913f7b273591f851 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:46.391675 10942 raft_consensus.cc:686] T 6488dcdd5ab04e35bbd720f4985ceb41 P a885359a82bd4515913f7b273591f851 [term 1 LEADER]: Becoming Leader. State: Replica: a885359a82bd4515913f7b273591f851, State: Running, Role: LEADER
I20240502 01:52:46.392037 10942 consensus_queue.cc:237] T 6488dcdd5ab04e35bbd720f4985ceb41 P a885359a82bd4515913f7b273591f851 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } }
I20240502 01:52:46.395193  9958 catalog_manager.cc:5484] T 6488dcdd5ab04e35bbd720f4985ceb41 P a885359a82bd4515913f7b273591f851 reported cstate change: term changed from 0 to 1, leader changed from <none> to a885359a82bd4515913f7b273591f851 (127.9.176.193). New cstate: current_term: 1 leader_uuid: "a885359a82bd4515913f7b273591f851" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } health_report { overall_health: UNKNOWN } } }
I20240502 01:52:46.766638 10942 consensus_queue.cc:1035] T de6e1081d71641248b0657bbdb824af8 P a885359a82bd4515913f7b273591f851 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:46.776265 10942 consensus_queue.cc:1035] T de6e1081d71641248b0657bbdb824af8 P a885359a82bd4515913f7b273591f851 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:52:46.805362 11450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:46.805745 11450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:46.813525 11450 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 01:52:46.846246 11339 consensus_queue.cc:1035] T 09a892cf118d41d4bfffc1576731d54c P fd95bcd4cdcb4cc5966050ffe2cffdff [LEADER]: Connected to new peer: Peer: permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_6'
I20240502 01:52:46.849155 11446 consensus_queue.cc:1035] T 6488dcdd5ab04e35bbd720f4985ceb41 P a885359a82bd4515913f7b273591f851 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:46.864255 11453 consensus_queue.cc:1035] T 6488dcdd5ab04e35bbd720f4985ceb41 P a885359a82bd4515913f7b273591f851 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:46.894217 11477 consensus_queue.cc:1035] T 09a892cf118d41d4bfffc1576731d54c P fd95bcd4cdcb4cc5966050ffe2cffdff [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 299.183 ms
  time per row: 0.149592 ms
-----------------------------------------------------
Suppressions used:
  count      bytes template
     14       3624 libcrypto.so.1.1
-----------------------------------------------------

I20240502 01:52:48.273555  9966 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56524:
name: "rebalance_test_table_7"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 01:52:48.283447 10492 tablet_service.cc:1467] Processing CreateTablet for tablet c2730c626fe34084ab71d463dd2b8656 (DEFAULT_TABLE table=rebalance_test_table_7 [id=745afd3765b14f5999e28c70c9d502a6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:48.283743 10758 tablet_service.cc:1467] Processing CreateTablet for tablet ad11d50337ab43fdbc110dc200035fd1 (DEFAULT_TABLE table=rebalance_test_table_7 [id=745afd3765b14f5999e28c70c9d502a6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:48.284034 10492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2730c626fe34084ab71d463dd2b8656. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:48.284196 10758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad11d50337ab43fdbc110dc200035fd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:48.284952 10759 tablet_service.cc:1467] Processing CreateTablet for tablet c2730c626fe34084ab71d463dd2b8656 (DEFAULT_TABLE table=rebalance_test_table_7 [id=745afd3765b14f5999e28c70c9d502a6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:48.284955 10225 tablet_service.cc:1467] Processing CreateTablet for tablet c2730c626fe34084ab71d463dd2b8656 (DEFAULT_TABLE table=rebalance_test_table_7 [id=745afd3765b14f5999e28c70c9d502a6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:48.285384 10225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2730c626fe34084ab71d463dd2b8656. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:48.285383 10759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2730c626fe34084ab71d463dd2b8656. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:48.286651 10092 tablet_service.cc:1467] Processing CreateTablet for tablet 941430c4941d4ce7bf4252f0b56abecc (DEFAULT_TABLE table=rebalance_test_table_7 [id=745afd3765b14f5999e28c70c9d502a6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:48.286669 10224 tablet_service.cc:1467] Processing CreateTablet for tablet ad11d50337ab43fdbc110dc200035fd1 (DEFAULT_TABLE table=rebalance_test_table_7 [id=745afd3765b14f5999e28c70c9d502a6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:48.287071 10092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 941430c4941d4ce7bf4252f0b56abecc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:48.287077 10224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad11d50337ab43fdbc110dc200035fd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:48.288630 10623 tablet_service.cc:1467] Processing CreateTablet for tablet 941430c4941d4ce7bf4252f0b56abecc (DEFAULT_TABLE table=rebalance_test_table_7 [id=745afd3765b14f5999e28c70c9d502a6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:48.288890 10491 tablet_service.cc:1467] Processing CreateTablet for tablet ad11d50337ab43fdbc110dc200035fd1 (DEFAULT_TABLE table=rebalance_test_table_7 [id=745afd3765b14f5999e28c70c9d502a6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:48.289094 10623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 941430c4941d4ce7bf4252f0b56abecc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:48.289350 10491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad11d50337ab43fdbc110dc200035fd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:48.292899 10357 tablet_service.cc:1467] Processing CreateTablet for tablet 941430c4941d4ce7bf4252f0b56abecc (DEFAULT_TABLE table=rebalance_test_table_7 [id=745afd3765b14f5999e28c70c9d502a6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:48.293378 10357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 941430c4941d4ce7bf4252f0b56abecc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:48.301434 11528 tablet_bootstrap.cc:492] T 941430c4941d4ce7bf4252f0b56abecc P a8c4a518929d4bb3a88bdfb1649493bd: Bootstrap starting.
I20240502 01:52:48.303171 11526 tablet_bootstrap.cc:492] T c2730c626fe34084ab71d463dd2b8656 P 34da849acb2045a9a2ef03933592dd08: Bootstrap starting.
I20240502 01:52:48.303552 11525 tablet_bootstrap.cc:492] T c2730c626fe34084ab71d463dd2b8656 P fd95bcd4cdcb4cc5966050ffe2cffdff: Bootstrap starting.
I20240502 01:52:48.304443 11528 tablet_bootstrap.cc:654] T 941430c4941d4ce7bf4252f0b56abecc P a8c4a518929d4bb3a88bdfb1649493bd: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:48.305431 11525 tablet_bootstrap.cc:654] T c2730c626fe34084ab71d463dd2b8656 P fd95bcd4cdcb4cc5966050ffe2cffdff: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:48.305564 11526 tablet_bootstrap.cc:654] T c2730c626fe34084ab71d463dd2b8656 P 34da849acb2045a9a2ef03933592dd08: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:48.313047 11528 tablet_bootstrap.cc:492] T 941430c4941d4ce7bf4252f0b56abecc P a8c4a518929d4bb3a88bdfb1649493bd: No bootstrap required, opened a new log
I20240502 01:52:48.313395 11528 ts_tablet_manager.cc:1367] T 941430c4941d4ce7bf4252f0b56abecc P a8c4a518929d4bb3a88bdfb1649493bd: Time spent bootstrapping tablet: real 0.012s	user 0.005s	sys 0.005s
I20240502 01:52:48.314563 11528 raft_consensus.cc:348] T 941430c4941d4ce7bf4252f0b56abecc P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } }
I20240502 01:52:48.315026 11528 raft_consensus.cc:374] T 941430c4941d4ce7bf4252f0b56abecc P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:48.315196 11528 raft_consensus.cc:729] T 941430c4941d4ce7bf4252f0b56abecc P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8c4a518929d4bb3a88bdfb1649493bd, State: Initialized, Role: FOLLOWER
I20240502 01:52:48.315670 11528 consensus_queue.cc:260] T 941430c4941d4ce7bf4252f0b56abecc P a8c4a518929d4bb3a88bdfb1649493bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } }
I20240502 01:52:48.316807 11528 ts_tablet_manager.cc:1398] T 941430c4941d4ce7bf4252f0b56abecc P a8c4a518929d4bb3a88bdfb1649493bd: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 01:52:48.318607 11525 tablet_bootstrap.cc:492] T c2730c626fe34084ab71d463dd2b8656 P fd95bcd4cdcb4cc5966050ffe2cffdff: No bootstrap required, opened a new log
I20240502 01:52:48.318859 11525 ts_tablet_manager.cc:1367] T c2730c626fe34084ab71d463dd2b8656 P fd95bcd4cdcb4cc5966050ffe2cffdff: Time spent bootstrapping tablet: real 0.015s	user 0.007s	sys 0.007s
I20240502 01:52:48.320071 11525 raft_consensus.cc:348] T c2730c626fe34084ab71d463dd2b8656 P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } } peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } }
I20240502 01:52:48.320372 11525 raft_consensus.cc:374] T c2730c626fe34084ab71d463dd2b8656 P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:48.320526 11525 raft_consensus.cc:729] T c2730c626fe34084ab71d463dd2b8656 P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd95bcd4cdcb4cc5966050ffe2cffdff, State: Initialized, Role: FOLLOWER
I20240502 01:52:48.320806 11525 consensus_queue.cc:260] T c2730c626fe34084ab71d463dd2b8656 P fd95bcd4cdcb4cc5966050ffe2cffdff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } } peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } }
I20240502 01:52:48.321491 11525 ts_tablet_manager.cc:1398] T c2730c626fe34084ab71d463dd2b8656 P fd95bcd4cdcb4cc5966050ffe2cffdff: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 01:52:48.321879 11525 tablet_bootstrap.cc:492] T ad11d50337ab43fdbc110dc200035fd1 P fd95bcd4cdcb4cc5966050ffe2cffdff: Bootstrap starting.
I20240502 01:52:48.323700 11525 tablet_bootstrap.cc:654] T ad11d50337ab43fdbc110dc200035fd1 P fd95bcd4cdcb4cc5966050ffe2cffdff: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:48.336390 11526 tablet_bootstrap.cc:492] T c2730c626fe34084ab71d463dd2b8656 P 34da849acb2045a9a2ef03933592dd08: No bootstrap required, opened a new log
I20240502 01:52:48.337229 11526 ts_tablet_manager.cc:1367] T c2730c626fe34084ab71d463dd2b8656 P 34da849acb2045a9a2ef03933592dd08: Time spent bootstrapping tablet: real 0.034s	user 0.004s	sys 0.009s
I20240502 01:52:48.338387 11527 tablet_bootstrap.cc:492] T 941430c4941d4ce7bf4252f0b56abecc P a885359a82bd4515913f7b273591f851: Bootstrap starting.
I20240502 01:52:48.338620 11535 tablet_bootstrap.cc:492] T c2730c626fe34084ab71d463dd2b8656 P 7b7a47245f294ce9a264fcc8f6351c0f: Bootstrap starting.
I20240502 01:52:48.339010 11526 raft_consensus.cc:348] T c2730c626fe34084ab71d463dd2b8656 P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } } peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } }
I20240502 01:52:48.339846 11533 tablet_bootstrap.cc:492] T 941430c4941d4ce7bf4252f0b56abecc P ede1dd598e3541138a19a3ea5c052bc9: Bootstrap starting.
I20240502 01:52:48.340032 11526 raft_consensus.cc:374] T c2730c626fe34084ab71d463dd2b8656 P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:48.340204 11526 raft_consensus.cc:729] T c2730c626fe34084ab71d463dd2b8656 P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34da849acb2045a9a2ef03933592dd08, State: Initialized, Role: FOLLOWER
I20240502 01:52:48.340689 11525 tablet_bootstrap.cc:492] T ad11d50337ab43fdbc110dc200035fd1 P fd95bcd4cdcb4cc5966050ffe2cffdff: No bootstrap required, opened a new log
I20240502 01:52:48.340596 11526 consensus_queue.cc:260] T c2730c626fe34084ab71d463dd2b8656 P 34da849acb2045a9a2ef03933592dd08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } } peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } }
I20240502 01:52:48.340754 11527 tablet_bootstrap.cc:654] T 941430c4941d4ce7bf4252f0b56abecc P a885359a82bd4515913f7b273591f851: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:48.340979 11525 ts_tablet_manager.cc:1367] T ad11d50337ab43fdbc110dc200035fd1 P fd95bcd4cdcb4cc5966050ffe2cffdff: Time spent bootstrapping tablet: real 0.019s	user 0.008s	sys 0.010s
I20240502 01:52:48.341975 11525 raft_consensus.cc:348] T ad11d50337ab43fdbc110dc200035fd1 P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } }
I20240502 01:52:48.342340 11525 raft_consensus.cc:374] T ad11d50337ab43fdbc110dc200035fd1 P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:48.342500 11533 tablet_bootstrap.cc:654] T 941430c4941d4ce7bf4252f0b56abecc P ede1dd598e3541138a19a3ea5c052bc9: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:48.342496 11525 raft_consensus.cc:729] T ad11d50337ab43fdbc110dc200035fd1 P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd95bcd4cdcb4cc5966050ffe2cffdff, State: Initialized, Role: FOLLOWER
I20240502 01:52:48.342765 11526 ts_tablet_manager.cc:1398] T c2730c626fe34084ab71d463dd2b8656 P 34da849acb2045a9a2ef03933592dd08: Time spent starting tablet: real 0.005s	user 0.001s	sys 0.001s
I20240502 01:52:48.342820 11525 consensus_queue.cc:260] T ad11d50337ab43fdbc110dc200035fd1 P fd95bcd4cdcb4cc5966050ffe2cffdff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } }
I20240502 01:52:48.343689 11525 ts_tablet_manager.cc:1398] T ad11d50337ab43fdbc110dc200035fd1 P fd95bcd4cdcb4cc5966050ffe2cffdff: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.001s
I20240502 01:52:48.345861 11535 tablet_bootstrap.cc:654] T c2730c626fe34084ab71d463dd2b8656 P 7b7a47245f294ce9a264fcc8f6351c0f: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:48.349380 11533 tablet_bootstrap.cc:492] T 941430c4941d4ce7bf4252f0b56abecc P ede1dd598e3541138a19a3ea5c052bc9: No bootstrap required, opened a new log
I20240502 01:52:48.349758 11533 ts_tablet_manager.cc:1367] T 941430c4941d4ce7bf4252f0b56abecc P ede1dd598e3541138a19a3ea5c052bc9: Time spent bootstrapping tablet: real 0.010s	user 0.007s	sys 0.000s
I20240502 01:52:48.350759 11533 raft_consensus.cc:348] T 941430c4941d4ce7bf4252f0b56abecc P ede1dd598e3541138a19a3ea5c052bc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } }
I20240502 01:52:48.351120 11533 raft_consensus.cc:374] T 941430c4941d4ce7bf4252f0b56abecc P ede1dd598e3541138a19a3ea5c052bc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:48.351285 11533 raft_consensus.cc:729] T 941430c4941d4ce7bf4252f0b56abecc P ede1dd598e3541138a19a3ea5c052bc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ede1dd598e3541138a19a3ea5c052bc9, State: Initialized, Role: FOLLOWER
I20240502 01:52:48.343207 11526 tablet_bootstrap.cc:492] T ad11d50337ab43fdbc110dc200035fd1 P 34da849acb2045a9a2ef03933592dd08: Bootstrap starting.
I20240502 01:52:48.352051 11533 consensus_queue.cc:260] T 941430c4941d4ce7bf4252f0b56abecc P ede1dd598e3541138a19a3ea5c052bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } }
I20240502 01:52:48.354126 11526 tablet_bootstrap.cc:654] T ad11d50337ab43fdbc110dc200035fd1 P 34da849acb2045a9a2ef03933592dd08: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:48.355600 11533 ts_tablet_manager.cc:1398] T 941430c4941d4ce7bf4252f0b56abecc P ede1dd598e3541138a19a3ea5c052bc9: Time spent starting tablet: real 0.006s	user 0.002s	sys 0.000s
I20240502 01:52:48.358192 11527 tablet_bootstrap.cc:492] T 941430c4941d4ce7bf4252f0b56abecc P a885359a82bd4515913f7b273591f851: No bootstrap required, opened a new log
I20240502 01:52:48.358453 11527 ts_tablet_manager.cc:1367] T 941430c4941d4ce7bf4252f0b56abecc P a885359a82bd4515913f7b273591f851: Time spent bootstrapping tablet: real 0.020s	user 0.004s	sys 0.010s
I20240502 01:52:48.359519 11527 raft_consensus.cc:348] T 941430c4941d4ce7bf4252f0b56abecc P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } }
I20240502 01:52:48.359869 11527 raft_consensus.cc:374] T 941430c4941d4ce7bf4252f0b56abecc P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:48.360039 11527 raft_consensus.cc:729] T 941430c4941d4ce7bf4252f0b56abecc P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a885359a82bd4515913f7b273591f851, State: Initialized, Role: FOLLOWER
I20240502 01:52:48.360365 11527 consensus_queue.cc:260] T 941430c4941d4ce7bf4252f0b56abecc P a885359a82bd4515913f7b273591f851 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } }
I20240502 01:52:48.361097 11527 ts_tablet_manager.cc:1398] T 941430c4941d4ce7bf4252f0b56abecc P a885359a82bd4515913f7b273591f851: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 01:52:48.362674 11535 tablet_bootstrap.cc:492] T c2730c626fe34084ab71d463dd2b8656 P 7b7a47245f294ce9a264fcc8f6351c0f: No bootstrap required, opened a new log
I20240502 01:52:48.362816 11526 tablet_bootstrap.cc:492] T ad11d50337ab43fdbc110dc200035fd1 P 34da849acb2045a9a2ef03933592dd08: No bootstrap required, opened a new log
I20240502 01:52:48.362855 11535 ts_tablet_manager.cc:1367] T c2730c626fe34084ab71d463dd2b8656 P 7b7a47245f294ce9a264fcc8f6351c0f: Time spent bootstrapping tablet: real 0.024s	user 0.007s	sys 0.008s
I20240502 01:52:48.363029 11526 ts_tablet_manager.cc:1367] T ad11d50337ab43fdbc110dc200035fd1 P 34da849acb2045a9a2ef03933592dd08: Time spent bootstrapping tablet: real 0.020s	user 0.001s	sys 0.008s
I20240502 01:52:48.363729 11535 raft_consensus.cc:348] T c2730c626fe34084ab71d463dd2b8656 P 7b7a47245f294ce9a264fcc8f6351c0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } } peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } }
I20240502 01:52:48.363971 11535 raft_consensus.cc:374] T c2730c626fe34084ab71d463dd2b8656 P 7b7a47245f294ce9a264fcc8f6351c0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:48.364109 11535 raft_consensus.cc:729] T c2730c626fe34084ab71d463dd2b8656 P 7b7a47245f294ce9a264fcc8f6351c0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b7a47245f294ce9a264fcc8f6351c0f, State: Initialized, Role: FOLLOWER
I20240502 01:52:48.364372 11535 consensus_queue.cc:260] T c2730c626fe34084ab71d463dd2b8656 P 7b7a47245f294ce9a264fcc8f6351c0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } } peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } }
I20240502 01:52:48.364490 11526 raft_consensus.cc:348] T ad11d50337ab43fdbc110dc200035fd1 P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } }
I20240502 01:52:48.364766 11526 raft_consensus.cc:374] T ad11d50337ab43fdbc110dc200035fd1 P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:48.364936 11526 raft_consensus.cc:729] T ad11d50337ab43fdbc110dc200035fd1 P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34da849acb2045a9a2ef03933592dd08, State: Initialized, Role: FOLLOWER
I20240502 01:52:48.365128 11535 ts_tablet_manager.cc:1398] T c2730c626fe34084ab71d463dd2b8656 P 7b7a47245f294ce9a264fcc8f6351c0f: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.000s
I20240502 01:52:48.365283 11526 consensus_queue.cc:260] T ad11d50337ab43fdbc110dc200035fd1 P 34da849acb2045a9a2ef03933592dd08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } }
I20240502 01:52:48.365494 11535 tablet_bootstrap.cc:492] T ad11d50337ab43fdbc110dc200035fd1 P 7b7a47245f294ce9a264fcc8f6351c0f: Bootstrap starting.
I20240502 01:52:48.366158 11526 ts_tablet_manager.cc:1398] T ad11d50337ab43fdbc110dc200035fd1 P 34da849acb2045a9a2ef03933592dd08: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.002s
I20240502 01:52:48.367341 11535 tablet_bootstrap.cc:654] T ad11d50337ab43fdbc110dc200035fd1 P 7b7a47245f294ce9a264fcc8f6351c0f: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:48.370587 11535 tablet_bootstrap.cc:492] T ad11d50337ab43fdbc110dc200035fd1 P 7b7a47245f294ce9a264fcc8f6351c0f: No bootstrap required, opened a new log
I20240502 01:52:48.371315 11535 ts_tablet_manager.cc:1367] T ad11d50337ab43fdbc110dc200035fd1 P 7b7a47245f294ce9a264fcc8f6351c0f: Time spent bootstrapping tablet: real 0.006s	user 0.004s	sys 0.000s
I20240502 01:52:48.372202 11535 raft_consensus.cc:348] T ad11d50337ab43fdbc110dc200035fd1 P 7b7a47245f294ce9a264fcc8f6351c0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } }
I20240502 01:52:48.372426 11535 raft_consensus.cc:374] T ad11d50337ab43fdbc110dc200035fd1 P 7b7a47245f294ce9a264fcc8f6351c0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:48.372496 11535 raft_consensus.cc:729] T ad11d50337ab43fdbc110dc200035fd1 P 7b7a47245f294ce9a264fcc8f6351c0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b7a47245f294ce9a264fcc8f6351c0f, State: Initialized, Role: FOLLOWER
I20240502 01:52:48.372696 11535 consensus_queue.cc:260] T ad11d50337ab43fdbc110dc200035fd1 P 7b7a47245f294ce9a264fcc8f6351c0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } }
I20240502 01:52:48.373183 11535 ts_tablet_manager.cc:1398] T ad11d50337ab43fdbc110dc200035fd1 P 7b7a47245f294ce9a264fcc8f6351c0f: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 01:52:48.434536 11173 raft_consensus.cc:482] T ad11d50337ab43fdbc110dc200035fd1 P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:48.434805 11173 raft_consensus.cc:504] T ad11d50337ab43fdbc110dc200035fd1 P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } }
I20240502 01:52:48.435547 11173 leader_election.cc:288] T ad11d50337ab43fdbc110dc200035fd1 P 34da849acb2045a9a2ef03933592dd08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fd95bcd4cdcb4cc5966050ffe2cffdff (127.9.176.198:33039), 7b7a47245f294ce9a264fcc8f6351c0f (127.9.176.194:43645)
I20240502 01:52:48.436082 10780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad11d50337ab43fdbc110dc200035fd1" candidate_uuid: "34da849acb2045a9a2ef03933592dd08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" is_pre_election: true
I20240502 01:52:48.436215 10245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad11d50337ab43fdbc110dc200035fd1" candidate_uuid: "34da849acb2045a9a2ef03933592dd08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" is_pre_election: true
I20240502 01:52:48.436343 10780 raft_consensus.cc:2454] T ad11d50337ab43fdbc110dc200035fd1 P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34da849acb2045a9a2ef03933592dd08 in term 0.
I20240502 01:52:48.436463 10245 raft_consensus.cc:2454] T ad11d50337ab43fdbc110dc200035fd1 P 7b7a47245f294ce9a264fcc8f6351c0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34da849acb2045a9a2ef03933592dd08 in term 0.
I20240502 01:52:48.436885 10445 leader_election.cc:302] T ad11d50337ab43fdbc110dc200035fd1 P 34da849acb2045a9a2ef03933592dd08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34da849acb2045a9a2ef03933592dd08, fd95bcd4cdcb4cc5966050ffe2cffdff; no voters: 
I20240502 01:52:48.437242 11173 raft_consensus.cc:2781] T ad11d50337ab43fdbc110dc200035fd1 P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:48.437350 11173 raft_consensus.cc:482] T ad11d50337ab43fdbc110dc200035fd1 P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:48.437472 11173 raft_consensus.cc:3037] T ad11d50337ab43fdbc110dc200035fd1 P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:48.439481 11173 raft_consensus.cc:504] T ad11d50337ab43fdbc110dc200035fd1 P 34da849acb2045a9a2ef03933592dd08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } }
I20240502 01:52:48.440268 11173 leader_election.cc:288] T ad11d50337ab43fdbc110dc200035fd1 P 34da849acb2045a9a2ef03933592dd08 [CANDIDATE]: Term 1 election: Requested vote from peers fd95bcd4cdcb4cc5966050ffe2cffdff (127.9.176.198:33039), 7b7a47245f294ce9a264fcc8f6351c0f (127.9.176.194:43645)
I20240502 01:52:48.440723 10780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad11d50337ab43fdbc110dc200035fd1" candidate_uuid: "34da849acb2045a9a2ef03933592dd08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff"
I20240502 01:52:48.440945 10780 raft_consensus.cc:3037] T ad11d50337ab43fdbc110dc200035fd1 P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:48.440886 10245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad11d50337ab43fdbc110dc200035fd1" candidate_uuid: "34da849acb2045a9a2ef03933592dd08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b7a47245f294ce9a264fcc8f6351c0f"
I20240502 01:52:48.441105 10245 raft_consensus.cc:3037] T ad11d50337ab43fdbc110dc200035fd1 P 7b7a47245f294ce9a264fcc8f6351c0f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:48.443177 10780 raft_consensus.cc:2454] T ad11d50337ab43fdbc110dc200035fd1 P fd95bcd4cdcb4cc5966050ffe2cffdff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34da849acb2045a9a2ef03933592dd08 in term 1.
I20240502 01:52:48.443193 10245 raft_consensus.cc:2454] T ad11d50337ab43fdbc110dc200035fd1 P 7b7a47245f294ce9a264fcc8f6351c0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34da849acb2045a9a2ef03933592dd08 in term 1.
I20240502 01:52:48.443732 10443 leader_election.cc:302] T ad11d50337ab43fdbc110dc200035fd1 P 34da849acb2045a9a2ef03933592dd08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34da849acb2045a9a2ef03933592dd08, 7b7a47245f294ce9a264fcc8f6351c0f; no voters: 
I20240502 01:52:48.444075 11173 raft_consensus.cc:2781] T ad11d50337ab43fdbc110dc200035fd1 P 34da849acb2045a9a2ef03933592dd08 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:48.444552 11173 raft_consensus.cc:686] T ad11d50337ab43fdbc110dc200035fd1 P 34da849acb2045a9a2ef03933592dd08 [term 1 LEADER]: Becoming Leader. State: Replica: 34da849acb2045a9a2ef03933592dd08, State: Running, Role: LEADER
I20240502 01:52:48.444887 11173 consensus_queue.cc:237] T ad11d50337ab43fdbc110dc200035fd1 P 34da849acb2045a9a2ef03933592dd08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } }
I20240502 01:52:48.447095 11178 raft_consensus.cc:482] T 941430c4941d4ce7bf4252f0b56abecc P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:48.447290 11178 raft_consensus.cc:504] T 941430c4941d4ce7bf4252f0b56abecc P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } }
I20240502 01:52:48.448072 11178 leader_election.cc:288] T 941430c4941d4ce7bf4252f0b56abecc P a8c4a518929d4bb3a88bdfb1649493bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a885359a82bd4515913f7b273591f851 (127.9.176.193:38373), ede1dd598e3541138a19a3ea5c052bc9 (127.9.176.195:38701)
I20240502 01:52:48.448524  9966 catalog_manager.cc:5484] T ad11d50337ab43fdbc110dc200035fd1 P 34da849acb2045a9a2ef03933592dd08 reported cstate change: term changed from 0 to 1, leader changed from <none> to 34da849acb2045a9a2ef03933592dd08 (127.9.176.196). New cstate: current_term: 1 leader_uuid: "34da849acb2045a9a2ef03933592dd08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } health_report { overall_health: HEALTHY } } }
I20240502 01:52:48.448894 10111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "941430c4941d4ce7bf4252f0b56abecc" candidate_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a885359a82bd4515913f7b273591f851" is_pre_election: true
I20240502 01:52:48.449108 10111 raft_consensus.cc:2454] T 941430c4941d4ce7bf4252f0b56abecc P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8c4a518929d4bb3a88bdfb1649493bd in term 0.
I20240502 01:52:48.449335 10378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "941430c4941d4ce7bf4252f0b56abecc" candidate_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ede1dd598e3541138a19a3ea5c052bc9" is_pre_election: true
I20240502 01:52:48.449600 10378 raft_consensus.cc:2454] T 941430c4941d4ce7bf4252f0b56abecc P ede1dd598e3541138a19a3ea5c052bc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8c4a518929d4bb3a88bdfb1649493bd in term 0.
I20240502 01:52:48.450006 10580 leader_election.cc:302] T 941430c4941d4ce7bf4252f0b56abecc P a8c4a518929d4bb3a88bdfb1649493bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a885359a82bd4515913f7b273591f851, a8c4a518929d4bb3a88bdfb1649493bd; no voters: 
I20240502 01:52:48.450706 11178 raft_consensus.cc:2781] T 941430c4941d4ce7bf4252f0b56abecc P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:48.450793 11178 raft_consensus.cc:482] T 941430c4941d4ce7bf4252f0b56abecc P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:48.450896 11178 raft_consensus.cc:3037] T 941430c4941d4ce7bf4252f0b56abecc P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:48.453191 11178 raft_consensus.cc:504] T 941430c4941d4ce7bf4252f0b56abecc P a8c4a518929d4bb3a88bdfb1649493bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } }
I20240502 01:52:48.453933 11178 leader_election.cc:288] T 941430c4941d4ce7bf4252f0b56abecc P a8c4a518929d4bb3a88bdfb1649493bd [CANDIDATE]: Term 1 election: Requested vote from peers a885359a82bd4515913f7b273591f851 (127.9.176.193:38373), ede1dd598e3541138a19a3ea5c052bc9 (127.9.176.195:38701)
I20240502 01:52:48.454447 10111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "941430c4941d4ce7bf4252f0b56abecc" candidate_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a885359a82bd4515913f7b273591f851"
I20240502 01:52:48.454447 10378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "941430c4941d4ce7bf4252f0b56abecc" candidate_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ede1dd598e3541138a19a3ea5c052bc9"
I20240502 01:52:48.454679 10378 raft_consensus.cc:3037] T 941430c4941d4ce7bf4252f0b56abecc P ede1dd598e3541138a19a3ea5c052bc9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:48.454676 10111 raft_consensus.cc:3037] T 941430c4941d4ce7bf4252f0b56abecc P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:48.457708 10378 raft_consensus.cc:2454] T 941430c4941d4ce7bf4252f0b56abecc P ede1dd598e3541138a19a3ea5c052bc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8c4a518929d4bb3a88bdfb1649493bd in term 1.
I20240502 01:52:48.457708 10111 raft_consensus.cc:2454] T 941430c4941d4ce7bf4252f0b56abecc P a885359a82bd4515913f7b273591f851 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8c4a518929d4bb3a88bdfb1649493bd in term 1.
I20240502 01:52:48.458297 10580 leader_election.cc:302] T 941430c4941d4ce7bf4252f0b56abecc P a8c4a518929d4bb3a88bdfb1649493bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a885359a82bd4515913f7b273591f851, a8c4a518929d4bb3a88bdfb1649493bd; no voters: 
I20240502 01:52:48.458595 11178 raft_consensus.cc:2781] T 941430c4941d4ce7bf4252f0b56abecc P a8c4a518929d4bb3a88bdfb1649493bd [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:48.458756 11178 raft_consensus.cc:686] T 941430c4941d4ce7bf4252f0b56abecc P a8c4a518929d4bb3a88bdfb1649493bd [term 1 LEADER]: Becoming Leader. State: Replica: a8c4a518929d4bb3a88bdfb1649493bd, State: Running, Role: LEADER
I20240502 01:52:48.459039 11178 consensus_queue.cc:237] T 941430c4941d4ce7bf4252f0b56abecc P a8c4a518929d4bb3a88bdfb1649493bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } }
I20240502 01:52:48.462136  9966 catalog_manager.cc:5484] T 941430c4941d4ce7bf4252f0b56abecc P a8c4a518929d4bb3a88bdfb1649493bd reported cstate change: term changed from 0 to 1, leader changed from <none> to a8c4a518929d4bb3a88bdfb1649493bd (127.9.176.197). New cstate: current_term: 1 leader_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } health_report { overall_health: HEALTHY } } }
I20240502 01:52:48.492846 11173 raft_consensus.cc:482] T c2730c626fe34084ab71d463dd2b8656 P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:48.493032 11173 raft_consensus.cc:504] T c2730c626fe34084ab71d463dd2b8656 P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } } peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } }
I20240502 01:52:48.493701 11173 leader_election.cc:288] T c2730c626fe34084ab71d463dd2b8656 P 34da849acb2045a9a2ef03933592dd08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fd95bcd4cdcb4cc5966050ffe2cffdff (127.9.176.198:33039), 7b7a47245f294ce9a264fcc8f6351c0f (127.9.176.194:43645)
I20240502 01:52:48.494336 10245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2730c626fe34084ab71d463dd2b8656" candidate_uuid: "34da849acb2045a9a2ef03933592dd08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" is_pre_election: true
I20240502 01:52:48.494316 10780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2730c626fe34084ab71d463dd2b8656" candidate_uuid: "34da849acb2045a9a2ef03933592dd08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" is_pre_election: true
I20240502 01:52:48.494587 10780 raft_consensus.cc:2454] T c2730c626fe34084ab71d463dd2b8656 P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34da849acb2045a9a2ef03933592dd08 in term 0.
I20240502 01:52:48.494606 10245 raft_consensus.cc:2454] T c2730c626fe34084ab71d463dd2b8656 P 7b7a47245f294ce9a264fcc8f6351c0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34da849acb2045a9a2ef03933592dd08 in term 0.
I20240502 01:52:48.495045 10445 leader_election.cc:302] T c2730c626fe34084ab71d463dd2b8656 P 34da849acb2045a9a2ef03933592dd08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34da849acb2045a9a2ef03933592dd08, fd95bcd4cdcb4cc5966050ffe2cffdff; no voters: 
I20240502 01:52:48.495327 11173 raft_consensus.cc:2781] T c2730c626fe34084ab71d463dd2b8656 P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:48.495469 11173 raft_consensus.cc:482] T c2730c626fe34084ab71d463dd2b8656 P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:48.495594 11173 raft_consensus.cc:3037] T c2730c626fe34084ab71d463dd2b8656 P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:48.497740 11173 raft_consensus.cc:504] T c2730c626fe34084ab71d463dd2b8656 P 34da849acb2045a9a2ef03933592dd08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } } peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } }
I20240502 01:52:48.498452 11173 leader_election.cc:288] T c2730c626fe34084ab71d463dd2b8656 P 34da849acb2045a9a2ef03933592dd08 [CANDIDATE]: Term 1 election: Requested vote from peers fd95bcd4cdcb4cc5966050ffe2cffdff (127.9.176.198:33039), 7b7a47245f294ce9a264fcc8f6351c0f (127.9.176.194:43645)
I20240502 01:52:48.499038 10245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2730c626fe34084ab71d463dd2b8656" candidate_uuid: "34da849acb2045a9a2ef03933592dd08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b7a47245f294ce9a264fcc8f6351c0f"
I20240502 01:52:48.499018 10780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2730c626fe34084ab71d463dd2b8656" candidate_uuid: "34da849acb2045a9a2ef03933592dd08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff"
I20240502 01:52:48.499255 10780 raft_consensus.cc:3037] T c2730c626fe34084ab71d463dd2b8656 P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:48.499276 10245 raft_consensus.cc:3037] T c2730c626fe34084ab71d463dd2b8656 P 7b7a47245f294ce9a264fcc8f6351c0f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:48.502265 10245 raft_consensus.cc:2454] T c2730c626fe34084ab71d463dd2b8656 P 7b7a47245f294ce9a264fcc8f6351c0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34da849acb2045a9a2ef03933592dd08 in term 1.
I20240502 01:52:48.502279 10780 raft_consensus.cc:2454] T c2730c626fe34084ab71d463dd2b8656 P fd95bcd4cdcb4cc5966050ffe2cffdff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34da849acb2045a9a2ef03933592dd08 in term 1.
I20240502 01:52:48.502885 10443 leader_election.cc:302] T c2730c626fe34084ab71d463dd2b8656 P 34da849acb2045a9a2ef03933592dd08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34da849acb2045a9a2ef03933592dd08, 7b7a47245f294ce9a264fcc8f6351c0f; no voters: 
I20240502 01:52:48.503248 11173 raft_consensus.cc:2781] T c2730c626fe34084ab71d463dd2b8656 P 34da849acb2045a9a2ef03933592dd08 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:48.503445 11173 raft_consensus.cc:686] T c2730c626fe34084ab71d463dd2b8656 P 34da849acb2045a9a2ef03933592dd08 [term 1 LEADER]: Becoming Leader. State: Replica: 34da849acb2045a9a2ef03933592dd08, State: Running, Role: LEADER
I20240502 01:52:48.503762 11173 consensus_queue.cc:237] T c2730c626fe34084ab71d463dd2b8656 P 34da849acb2045a9a2ef03933592dd08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } } peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } }
I20240502 01:52:48.506992  9966 catalog_manager.cc:5484] T c2730c626fe34084ab71d463dd2b8656 P 34da849acb2045a9a2ef03933592dd08 reported cstate change: term changed from 0 to 1, leader changed from <none> to 34da849acb2045a9a2ef03933592dd08 (127.9.176.196). New cstate: current_term: 1 leader_uuid: "34da849acb2045a9a2ef03933592dd08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } health_report { overall_health: UNKNOWN } } }
W20240502 01:52:48.923825 11546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:48.924325 11546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:48.932332 11546 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 01:52:48.966239 11173 consensus_queue.cc:1035] T ad11d50337ab43fdbc110dc200035fd1 P 34da849acb2045a9a2ef03933592dd08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_7'
I20240502 01:52:48.976958 11541 consensus_queue.cc:1035] T ad11d50337ab43fdbc110dc200035fd1 P 34da849acb2045a9a2ef03933592dd08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:48.978991 11178 consensus_queue.cc:1035] T 941430c4941d4ce7bf4252f0b56abecc P a8c4a518929d4bb3a88bdfb1649493bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:48.994122 11565 consensus_queue.cc:1035] T c2730c626fe34084ab71d463dd2b8656 P 34da849acb2045a9a2ef03933592dd08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:49.004808 11178 consensus_queue.cc:1035] T 941430c4941d4ce7bf4252f0b56abecc P a8c4a518929d4bb3a88bdfb1649493bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:49.004874 11173 consensus_queue.cc:1035] T c2730c626fe34084ab71d463dd2b8656 P 34da849acb2045a9a2ef03933592dd08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 258.343 ms
  time per row: 0.129172 ms
I20240502 01:52:49.399842  9967 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56524:
name: "rebalance_test_table_8"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 01:52:49.409168 10357 tablet_service.cc:1467] Processing CreateTablet for tablet c9f395cabd05425484aad0f82b0a3ca9 (DEFAULT_TABLE table=rebalance_test_table_8 [id=710b65d7c885428d8c7dea476bc8ff0e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:49.409354 10623 tablet_service.cc:1467] Processing CreateTablet for tablet c9f395cabd05425484aad0f82b0a3ca9 (DEFAULT_TABLE table=rebalance_test_table_8 [id=710b65d7c885428d8c7dea476bc8ff0e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:49.409703 10357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9f395cabd05425484aad0f82b0a3ca9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:49.409757 10623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9f395cabd05425484aad0f82b0a3ca9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:49.410017 10092 tablet_service.cc:1467] Processing CreateTablet for tablet c9f395cabd05425484aad0f82b0a3ca9 (DEFAULT_TABLE table=rebalance_test_table_8 [id=710b65d7c885428d8c7dea476bc8ff0e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:49.410444 10092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9f395cabd05425484aad0f82b0a3ca9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:49.410697 10624 tablet_service.cc:1467] Processing CreateTablet for tablet 673fc226d6b441219d1153741b914808 (DEFAULT_TABLE table=rebalance_test_table_8 [id=710b65d7c885428d8c7dea476bc8ff0e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:49.411082 10624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 673fc226d6b441219d1153741b914808. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:49.416021 10225 tablet_service.cc:1467] Processing CreateTablet for tablet 3a4f610cdc9d4dd48fd44924e406d929 (DEFAULT_TABLE table=rebalance_test_table_8 [id=710b65d7c885428d8c7dea476bc8ff0e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:49.416095 10091 tablet_service.cc:1467] Processing CreateTablet for tablet 673fc226d6b441219d1153741b914808 (DEFAULT_TABLE table=rebalance_test_table_8 [id=710b65d7c885428d8c7dea476bc8ff0e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:49.416534 10091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 673fc226d6b441219d1153741b914808. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:49.416642 10225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a4f610cdc9d4dd48fd44924e406d929. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:49.420655 10491 tablet_service.cc:1467] Processing CreateTablet for tablet 3a4f610cdc9d4dd48fd44924e406d929 (DEFAULT_TABLE table=rebalance_test_table_8 [id=710b65d7c885428d8c7dea476bc8ff0e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:49.421265 10491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a4f610cdc9d4dd48fd44924e406d929. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:49.424115 10759 tablet_service.cc:1467] Processing CreateTablet for tablet 3a4f610cdc9d4dd48fd44924e406d929 (DEFAULT_TABLE table=rebalance_test_table_8 [id=710b65d7c885428d8c7dea476bc8ff0e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:49.424657 10759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a4f610cdc9d4dd48fd44924e406d929. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:49.425956 11618 tablet_bootstrap.cc:492] T c9f395cabd05425484aad0f82b0a3ca9 P a885359a82bd4515913f7b273591f851: Bootstrap starting.
I20240502 01:52:49.428524 11618 tablet_bootstrap.cc:654] T c9f395cabd05425484aad0f82b0a3ca9 P a885359a82bd4515913f7b273591f851: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:49.428537 10358 tablet_service.cc:1467] Processing CreateTablet for tablet 673fc226d6b441219d1153741b914808 (DEFAULT_TABLE table=rebalance_test_table_8 [id=710b65d7c885428d8c7dea476bc8ff0e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:49.429046 10358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 673fc226d6b441219d1153741b914808. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:49.438302 11618 tablet_bootstrap.cc:492] T c9f395cabd05425484aad0f82b0a3ca9 P a885359a82bd4515913f7b273591f851: No bootstrap required, opened a new log
I20240502 01:52:49.438551 11618 ts_tablet_manager.cc:1367] T c9f395cabd05425484aad0f82b0a3ca9 P a885359a82bd4515913f7b273591f851: Time spent bootstrapping tablet: real 0.013s	user 0.005s	sys 0.005s
I20240502 01:52:49.439460 11618 raft_consensus.cc:348] T c9f395cabd05425484aad0f82b0a3ca9 P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } }
I20240502 01:52:49.439783 11618 raft_consensus.cc:374] T c9f395cabd05425484aad0f82b0a3ca9 P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:49.439918 11618 raft_consensus.cc:729] T c9f395cabd05425484aad0f82b0a3ca9 P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a885359a82bd4515913f7b273591f851, State: Initialized, Role: FOLLOWER
I20240502 01:52:49.440258 11618 consensus_queue.cc:260] T c9f395cabd05425484aad0f82b0a3ca9 P a885359a82bd4515913f7b273591f851 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } }
I20240502 01:52:49.441093 11618 ts_tablet_manager.cc:1398] T c9f395cabd05425484aad0f82b0a3ca9 P a885359a82bd4515913f7b273591f851: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 01:52:49.442736 11618 tablet_bootstrap.cc:492] T 673fc226d6b441219d1153741b914808 P a885359a82bd4515913f7b273591f851: Bootstrap starting.
I20240502 01:52:49.444680 11618 tablet_bootstrap.cc:654] T 673fc226d6b441219d1153741b914808 P a885359a82bd4515913f7b273591f851: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:49.447338 11620 tablet_bootstrap.cc:492] T 673fc226d6b441219d1153741b914808 P ede1dd598e3541138a19a3ea5c052bc9: Bootstrap starting.
I20240502 01:52:49.449860 11622 tablet_bootstrap.cc:492] T 3a4f610cdc9d4dd48fd44924e406d929 P fd95bcd4cdcb4cc5966050ffe2cffdff: Bootstrap starting.
I20240502 01:52:49.450462 11620 tablet_bootstrap.cc:654] T 673fc226d6b441219d1153741b914808 P ede1dd598e3541138a19a3ea5c052bc9: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:49.452893 11622 tablet_bootstrap.cc:654] T 3a4f610cdc9d4dd48fd44924e406d929 P fd95bcd4cdcb4cc5966050ffe2cffdff: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:49.453637 11617 tablet_bootstrap.cc:492] T c9f395cabd05425484aad0f82b0a3ca9 P a8c4a518929d4bb3a88bdfb1649493bd: Bootstrap starting.
I20240502 01:52:49.452750 11624 tablet_bootstrap.cc:492] T 3a4f610cdc9d4dd48fd44924e406d929 P 7b7a47245f294ce9a264fcc8f6351c0f: Bootstrap starting.
I20240502 01:52:49.456468 11617 tablet_bootstrap.cc:654] T c9f395cabd05425484aad0f82b0a3ca9 P a8c4a518929d4bb3a88bdfb1649493bd: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:49.457186 11618 tablet_bootstrap.cc:492] T 673fc226d6b441219d1153741b914808 P a885359a82bd4515913f7b273591f851: No bootstrap required, opened a new log
I20240502 01:52:49.457837 11618 ts_tablet_manager.cc:1367] T 673fc226d6b441219d1153741b914808 P a885359a82bd4515913f7b273591f851: Time spent bootstrapping tablet: real 0.015s	user 0.000s	sys 0.010s
I20240502 01:52:49.458985 11623 tablet_bootstrap.cc:492] T 3a4f610cdc9d4dd48fd44924e406d929 P 34da849acb2045a9a2ef03933592dd08: Bootstrap starting.
I20240502 01:52:49.458987 11618 raft_consensus.cc:348] T 673fc226d6b441219d1153741b914808 P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } }
I20240502 01:52:49.459252 11618 raft_consensus.cc:374] T 673fc226d6b441219d1153741b914808 P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:49.459352 11618 raft_consensus.cc:729] T 673fc226d6b441219d1153741b914808 P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a885359a82bd4515913f7b273591f851, State: Initialized, Role: FOLLOWER
I20240502 01:52:49.459643 11622 tablet_bootstrap.cc:492] T 3a4f610cdc9d4dd48fd44924e406d929 P fd95bcd4cdcb4cc5966050ffe2cffdff: No bootstrap required, opened a new log
I20240502 01:52:49.459681 11618 consensus_queue.cc:260] T 673fc226d6b441219d1153741b914808 P a885359a82bd4515913f7b273591f851 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } }
I20240502 01:52:49.460273 11622 ts_tablet_manager.cc:1367] T 3a4f610cdc9d4dd48fd44924e406d929 P fd95bcd4cdcb4cc5966050ffe2cffdff: Time spent bootstrapping tablet: real 0.012s	user 0.006s	sys 0.000s
I20240502 01:52:49.460322 11618 ts_tablet_manager.cc:1398] T 673fc226d6b441219d1153741b914808 P a885359a82bd4515913f7b273591f851: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240502 01:52:49.461531 11622 raft_consensus.cc:348] T 3a4f610cdc9d4dd48fd44924e406d929 P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } } peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } }
I20240502 01:52:49.461937 11624 tablet_bootstrap.cc:654] T 3a4f610cdc9d4dd48fd44924e406d929 P 7b7a47245f294ce9a264fcc8f6351c0f: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:49.461963 11622 raft_consensus.cc:374] T 3a4f610cdc9d4dd48fd44924e406d929 P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:49.462200 11622 raft_consensus.cc:729] T 3a4f610cdc9d4dd48fd44924e406d929 P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd95bcd4cdcb4cc5966050ffe2cffdff, State: Initialized, Role: FOLLOWER
I20240502 01:52:49.462652 11622 consensus_queue.cc:260] T 3a4f610cdc9d4dd48fd44924e406d929 P fd95bcd4cdcb4cc5966050ffe2cffdff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } } peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } }
I20240502 01:52:49.463614 11622 ts_tablet_manager.cc:1398] T 3a4f610cdc9d4dd48fd44924e406d929 P fd95bcd4cdcb4cc5966050ffe2cffdff: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 01:52:49.465556 11617 tablet_bootstrap.cc:492] T c9f395cabd05425484aad0f82b0a3ca9 P a8c4a518929d4bb3a88bdfb1649493bd: No bootstrap required, opened a new log
I20240502 01:52:49.465682 11624 tablet_bootstrap.cc:492] T 3a4f610cdc9d4dd48fd44924e406d929 P 7b7a47245f294ce9a264fcc8f6351c0f: No bootstrap required, opened a new log
I20240502 01:52:49.465763 11617 ts_tablet_manager.cc:1367] T c9f395cabd05425484aad0f82b0a3ca9 P a8c4a518929d4bb3a88bdfb1649493bd: Time spent bootstrapping tablet: real 0.012s	user 0.005s	sys 0.005s
I20240502 01:52:49.465857 11624 ts_tablet_manager.cc:1367] T 3a4f610cdc9d4dd48fd44924e406d929 P 7b7a47245f294ce9a264fcc8f6351c0f: Time spent bootstrapping tablet: real 0.013s	user 0.002s	sys 0.002s
I20240502 01:52:49.466713 11617 raft_consensus.cc:348] T c9f395cabd05425484aad0f82b0a3ca9 P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } }
I20240502 01:52:49.467023 11617 raft_consensus.cc:374] T c9f395cabd05425484aad0f82b0a3ca9 P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:49.467182 11617 raft_consensus.cc:729] T c9f395cabd05425484aad0f82b0a3ca9 P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8c4a518929d4bb3a88bdfb1649493bd, State: Initialized, Role: FOLLOWER
I20240502 01:52:49.467128 11624 raft_consensus.cc:348] T 3a4f610cdc9d4dd48fd44924e406d929 P 7b7a47245f294ce9a264fcc8f6351c0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } } peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } }
I20240502 01:52:49.467496 11617 consensus_queue.cc:260] T c9f395cabd05425484aad0f82b0a3ca9 P a8c4a518929d4bb3a88bdfb1649493bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } }
I20240502 01:52:49.468379 11624 raft_consensus.cc:374] T 3a4f610cdc9d4dd48fd44924e406d929 P 7b7a47245f294ce9a264fcc8f6351c0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:49.468508 11624 raft_consensus.cc:729] T 3a4f610cdc9d4dd48fd44924e406d929 P 7b7a47245f294ce9a264fcc8f6351c0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b7a47245f294ce9a264fcc8f6351c0f, State: Initialized, Role: FOLLOWER
I20240502 01:52:49.468757 11624 consensus_queue.cc:260] T 3a4f610cdc9d4dd48fd44924e406d929 P 7b7a47245f294ce9a264fcc8f6351c0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } } peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } }
I20240502 01:52:49.469365 11624 ts_tablet_manager.cc:1398] T 3a4f610cdc9d4dd48fd44924e406d929 P 7b7a47245f294ce9a264fcc8f6351c0f: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.001s
I20240502 01:52:49.469770 11617 ts_tablet_manager.cc:1398] T c9f395cabd05425484aad0f82b0a3ca9 P a8c4a518929d4bb3a88bdfb1649493bd: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.000s
I20240502 01:52:49.470176 11617 tablet_bootstrap.cc:492] T 673fc226d6b441219d1153741b914808 P a8c4a518929d4bb3a88bdfb1649493bd: Bootstrap starting.
I20240502 01:52:49.470813 11620 tablet_bootstrap.cc:492] T 673fc226d6b441219d1153741b914808 P ede1dd598e3541138a19a3ea5c052bc9: No bootstrap required, opened a new log
I20240502 01:52:49.471022 11620 ts_tablet_manager.cc:1367] T 673fc226d6b441219d1153741b914808 P ede1dd598e3541138a19a3ea5c052bc9: Time spent bootstrapping tablet: real 0.024s	user 0.003s	sys 0.002s
I20240502 01:52:49.472195 11617 tablet_bootstrap.cc:654] T 673fc226d6b441219d1153741b914808 P a8c4a518929d4bb3a88bdfb1649493bd: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:49.473719 11620 raft_consensus.cc:348] T 673fc226d6b441219d1153741b914808 P ede1dd598e3541138a19a3ea5c052bc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } }
I20240502 01:52:49.474025 11620 raft_consensus.cc:374] T 673fc226d6b441219d1153741b914808 P ede1dd598e3541138a19a3ea5c052bc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:49.474160 11620 raft_consensus.cc:729] T 673fc226d6b441219d1153741b914808 P ede1dd598e3541138a19a3ea5c052bc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ede1dd598e3541138a19a3ea5c052bc9, State: Initialized, Role: FOLLOWER
I20240502 01:52:49.474507 11617 tablet_bootstrap.cc:492] T 673fc226d6b441219d1153741b914808 P a8c4a518929d4bb3a88bdfb1649493bd: No bootstrap required, opened a new log
I20240502 01:52:49.474473 11620 consensus_queue.cc:260] T 673fc226d6b441219d1153741b914808 P ede1dd598e3541138a19a3ea5c052bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } }
I20240502 01:52:49.474822 11617 ts_tablet_manager.cc:1367] T 673fc226d6b441219d1153741b914808 P a8c4a518929d4bb3a88bdfb1649493bd: Time spent bootstrapping tablet: real 0.005s	user 0.004s	sys 0.000s
I20240502 01:52:49.475931 11620 ts_tablet_manager.cc:1398] T 673fc226d6b441219d1153741b914808 P ede1dd598e3541138a19a3ea5c052bc9: Time spent starting tablet: real 0.005s	user 0.002s	sys 0.001s
I20240502 01:52:49.475807 11617 raft_consensus.cc:348] T 673fc226d6b441219d1153741b914808 P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } }
I20240502 01:52:49.476163 11617 raft_consensus.cc:374] T 673fc226d6b441219d1153741b914808 P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:49.476235 11620 tablet_bootstrap.cc:492] T c9f395cabd05425484aad0f82b0a3ca9 P ede1dd598e3541138a19a3ea5c052bc9: Bootstrap starting.
I20240502 01:52:49.476341 11617 raft_consensus.cc:729] T 673fc226d6b441219d1153741b914808 P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8c4a518929d4bb3a88bdfb1649493bd, State: Initialized, Role: FOLLOWER
I20240502 01:52:49.477190 11617 consensus_queue.cc:260] T 673fc226d6b441219d1153741b914808 P a8c4a518929d4bb3a88bdfb1649493bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } }
I20240502 01:52:49.477902 11620 tablet_bootstrap.cc:654] T c9f395cabd05425484aad0f82b0a3ca9 P ede1dd598e3541138a19a3ea5c052bc9: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:49.477900 11617 ts_tablet_manager.cc:1398] T 673fc226d6b441219d1153741b914808 P a8c4a518929d4bb3a88bdfb1649493bd: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 01:52:49.484622 11623 tablet_bootstrap.cc:654] T 3a4f610cdc9d4dd48fd44924e406d929 P 34da849acb2045a9a2ef03933592dd08: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:49.488395 11623 tablet_bootstrap.cc:492] T 3a4f610cdc9d4dd48fd44924e406d929 P 34da849acb2045a9a2ef03933592dd08: No bootstrap required, opened a new log
I20240502 01:52:49.489135 11623 ts_tablet_manager.cc:1367] T 3a4f610cdc9d4dd48fd44924e406d929 P 34da849acb2045a9a2ef03933592dd08: Time spent bootstrapping tablet: real 0.031s	user 0.003s	sys 0.001s
I20240502 01:52:49.489913 11623 raft_consensus.cc:348] T 3a4f610cdc9d4dd48fd44924e406d929 P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } } peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } }
I20240502 01:52:49.490108 11623 raft_consensus.cc:374] T 3a4f610cdc9d4dd48fd44924e406d929 P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:49.490202 11623 raft_consensus.cc:729] T 3a4f610cdc9d4dd48fd44924e406d929 P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34da849acb2045a9a2ef03933592dd08, State: Initialized, Role: FOLLOWER
I20240502 01:52:49.490548 11623 consensus_queue.cc:260] T 3a4f610cdc9d4dd48fd44924e406d929 P 34da849acb2045a9a2ef03933592dd08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } } peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } }
I20240502 01:52:49.491160 11623 ts_tablet_manager.cc:1398] T 3a4f610cdc9d4dd48fd44924e406d929 P 34da849acb2045a9a2ef03933592dd08: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240502 01:52:49.492683 11620 tablet_bootstrap.cc:492] T c9f395cabd05425484aad0f82b0a3ca9 P ede1dd598e3541138a19a3ea5c052bc9: No bootstrap required, opened a new log
I20240502 01:52:49.492884 11620 ts_tablet_manager.cc:1367] T c9f395cabd05425484aad0f82b0a3ca9 P ede1dd598e3541138a19a3ea5c052bc9: Time spent bootstrapping tablet: real 0.017s	user 0.009s	sys 0.005s
I20240502 01:52:49.493721 11620 raft_consensus.cc:348] T c9f395cabd05425484aad0f82b0a3ca9 P ede1dd598e3541138a19a3ea5c052bc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } }
I20240502 01:52:49.493968 11620 raft_consensus.cc:374] T c9f395cabd05425484aad0f82b0a3ca9 P ede1dd598e3541138a19a3ea5c052bc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:49.494046 11620 raft_consensus.cc:729] T c9f395cabd05425484aad0f82b0a3ca9 P ede1dd598e3541138a19a3ea5c052bc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ede1dd598e3541138a19a3ea5c052bc9, State: Initialized, Role: FOLLOWER
I20240502 01:52:49.494338 11620 consensus_queue.cc:260] T c9f395cabd05425484aad0f82b0a3ca9 P ede1dd598e3541138a19a3ea5c052bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } }
I20240502 01:52:49.495740 11620 ts_tablet_manager.cc:1398] T c9f395cabd05425484aad0f82b0a3ca9 P ede1dd598e3541138a19a3ea5c052bc9: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.001s
I20240502 01:52:49.497797 11574 raft_consensus.cc:482] T c9f395cabd05425484aad0f82b0a3ca9 P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:49.497967 11574 raft_consensus.cc:504] T c9f395cabd05425484aad0f82b0a3ca9 P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } }
I20240502 01:52:49.498792 11574 leader_election.cc:288] T c9f395cabd05425484aad0f82b0a3ca9 P a8c4a518929d4bb3a88bdfb1649493bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ede1dd598e3541138a19a3ea5c052bc9 (127.9.176.195:38701), a885359a82bd4515913f7b273591f851 (127.9.176.193:38373)
I20240502 01:52:49.499198 10377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9f395cabd05425484aad0f82b0a3ca9" candidate_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ede1dd598e3541138a19a3ea5c052bc9" is_pre_election: true
I20240502 01:52:49.499464 10377 raft_consensus.cc:2454] T c9f395cabd05425484aad0f82b0a3ca9 P ede1dd598e3541138a19a3ea5c052bc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8c4a518929d4bb3a88bdfb1649493bd in term 0.
I20240502 01:52:49.499493 10111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9f395cabd05425484aad0f82b0a3ca9" candidate_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a885359a82bd4515913f7b273591f851" is_pre_election: true
I20240502 01:52:49.499814 10111 raft_consensus.cc:2454] T c9f395cabd05425484aad0f82b0a3ca9 P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8c4a518929d4bb3a88bdfb1649493bd in term 0.
I20240502 01:52:49.500329 10580 leader_election.cc:302] T c9f395cabd05425484aad0f82b0a3ca9 P a8c4a518929d4bb3a88bdfb1649493bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a885359a82bd4515913f7b273591f851, a8c4a518929d4bb3a88bdfb1649493bd; no voters: 
I20240502 01:52:49.500592 11574 raft_consensus.cc:2781] T c9f395cabd05425484aad0f82b0a3ca9 P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:49.500702 11574 raft_consensus.cc:482] T c9f395cabd05425484aad0f82b0a3ca9 P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:49.500802 11574 raft_consensus.cc:3037] T c9f395cabd05425484aad0f82b0a3ca9 P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:49.502952 11574 raft_consensus.cc:504] T c9f395cabd05425484aad0f82b0a3ca9 P a8c4a518929d4bb3a88bdfb1649493bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } }
I20240502 01:52:49.503649 11446 raft_consensus.cc:482] T 673fc226d6b441219d1153741b914808 P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:49.503851 11446 raft_consensus.cc:504] T 673fc226d6b441219d1153741b914808 P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } }
I20240502 01:52:49.504034 11574 leader_election.cc:288] T c9f395cabd05425484aad0f82b0a3ca9 P a8c4a518929d4bb3a88bdfb1649493bd [CANDIDATE]: Term 1 election: Requested vote from peers ede1dd598e3541138a19a3ea5c052bc9 (127.9.176.195:38701), a885359a82bd4515913f7b273591f851 (127.9.176.193:38373)
I20240502 01:52:49.504010 10377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9f395cabd05425484aad0f82b0a3ca9" candidate_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ede1dd598e3541138a19a3ea5c052bc9"
I20240502 01:52:49.504247 10377 raft_consensus.cc:3037] T c9f395cabd05425484aad0f82b0a3ca9 P ede1dd598e3541138a19a3ea5c052bc9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:49.504566 11446 leader_election.cc:288] T 673fc226d6b441219d1153741b914808 P a885359a82bd4515913f7b273591f851 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a8c4a518929d4bb3a88bdfb1649493bd (127.9.176.197:44061), ede1dd598e3541138a19a3ea5c052bc9 (127.9.176.195:38701)
I20240502 01:52:49.504860 10111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9f395cabd05425484aad0f82b0a3ca9" candidate_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a885359a82bd4515913f7b273591f851"
I20240502 01:52:49.505059 10111 raft_consensus.cc:3037] T c9f395cabd05425484aad0f82b0a3ca9 P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:49.505107 10644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "673fc226d6b441219d1153741b914808" candidate_uuid: "a885359a82bd4515913f7b273591f851" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" is_pre_election: true
I20240502 01:52:49.505331 10644 raft_consensus.cc:2454] T 673fc226d6b441219d1153741b914808 P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a885359a82bd4515913f7b273591f851 in term 0.
I20240502 01:52:49.505837 10046 leader_election.cc:302] T 673fc226d6b441219d1153741b914808 P a885359a82bd4515913f7b273591f851 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a885359a82bd4515913f7b273591f851, a8c4a518929d4bb3a88bdfb1649493bd; no voters: 
I20240502 01:52:49.506209 11446 raft_consensus.cc:2781] T 673fc226d6b441219d1153741b914808 P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:49.506325 11446 raft_consensus.cc:482] T 673fc226d6b441219d1153741b914808 P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:49.506429 11446 raft_consensus.cc:3037] T 673fc226d6b441219d1153741b914808 P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:49.506717 10377 raft_consensus.cc:2454] T c9f395cabd05425484aad0f82b0a3ca9 P ede1dd598e3541138a19a3ea5c052bc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8c4a518929d4bb3a88bdfb1649493bd in term 1.
I20240502 01:52:49.507007 10378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "673fc226d6b441219d1153741b914808" candidate_uuid: "a885359a82bd4515913f7b273591f851" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ede1dd598e3541138a19a3ea5c052bc9" is_pre_election: true
I20240502 01:52:49.507200 10378 raft_consensus.cc:2454] T 673fc226d6b441219d1153741b914808 P ede1dd598e3541138a19a3ea5c052bc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a885359a82bd4515913f7b273591f851 in term 0.
I20240502 01:52:49.507494 10579 leader_election.cc:302] T c9f395cabd05425484aad0f82b0a3ca9 P a8c4a518929d4bb3a88bdfb1649493bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a8c4a518929d4bb3a88bdfb1649493bd, ede1dd598e3541138a19a3ea5c052bc9; no voters: 
I20240502 01:52:49.507493 10111 raft_consensus.cc:2454] T c9f395cabd05425484aad0f82b0a3ca9 P a885359a82bd4515913f7b273591f851 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8c4a518929d4bb3a88bdfb1649493bd in term 1.
I20240502 01:52:49.508024 11574 raft_consensus.cc:2781] T c9f395cabd05425484aad0f82b0a3ca9 P a8c4a518929d4bb3a88bdfb1649493bd [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:49.508189 11574 raft_consensus.cc:686] T c9f395cabd05425484aad0f82b0a3ca9 P a8c4a518929d4bb3a88bdfb1649493bd [term 1 LEADER]: Becoming Leader. State: Replica: a8c4a518929d4bb3a88bdfb1649493bd, State: Running, Role: LEADER
I20240502 01:52:49.508489 11574 consensus_queue.cc:237] T c9f395cabd05425484aad0f82b0a3ca9 P a8c4a518929d4bb3a88bdfb1649493bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } }
I20240502 01:52:49.511907  9967 catalog_manager.cc:5484] T c9f395cabd05425484aad0f82b0a3ca9 P a8c4a518929d4bb3a88bdfb1649493bd reported cstate change: term changed from 0 to 1, leader changed from <none> to a8c4a518929d4bb3a88bdfb1649493bd (127.9.176.197). New cstate: current_term: 1 leader_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } health_report { overall_health: UNKNOWN } } }
I20240502 01:52:49.526413 11446 raft_consensus.cc:504] T 673fc226d6b441219d1153741b914808 P a885359a82bd4515913f7b273591f851 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } }
I20240502 01:52:49.527267 11446 leader_election.cc:288] T 673fc226d6b441219d1153741b914808 P a885359a82bd4515913f7b273591f851 [CANDIDATE]: Term 1 election: Requested vote from peers a8c4a518929d4bb3a88bdfb1649493bd (127.9.176.197:44061), ede1dd598e3541138a19a3ea5c052bc9 (127.9.176.195:38701)
I20240502 01:52:49.527961 10644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "673fc226d6b441219d1153741b914808" candidate_uuid: "a885359a82bd4515913f7b273591f851" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8c4a518929d4bb3a88bdfb1649493bd"
I20240502 01:52:49.528095 10378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "673fc226d6b441219d1153741b914808" candidate_uuid: "a885359a82bd4515913f7b273591f851" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ede1dd598e3541138a19a3ea5c052bc9"
I20240502 01:52:49.528223 10644 raft_consensus.cc:3037] T 673fc226d6b441219d1153741b914808 P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:49.528370 10378 raft_consensus.cc:3037] T 673fc226d6b441219d1153741b914808 P ede1dd598e3541138a19a3ea5c052bc9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:49.530614 10378 raft_consensus.cc:2454] T 673fc226d6b441219d1153741b914808 P ede1dd598e3541138a19a3ea5c052bc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a885359a82bd4515913f7b273591f851 in term 1.
I20240502 01:52:49.530831 10644 raft_consensus.cc:2454] T 673fc226d6b441219d1153741b914808 P a8c4a518929d4bb3a88bdfb1649493bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a885359a82bd4515913f7b273591f851 in term 1.
I20240502 01:52:49.531078 10045 leader_election.cc:302] T 673fc226d6b441219d1153741b914808 P a885359a82bd4515913f7b273591f851 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a885359a82bd4515913f7b273591f851, ede1dd598e3541138a19a3ea5c052bc9; no voters: 
I20240502 01:52:49.531409 11446 raft_consensus.cc:2781] T 673fc226d6b441219d1153741b914808 P a885359a82bd4515913f7b273591f851 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:49.531589 11446 raft_consensus.cc:686] T 673fc226d6b441219d1153741b914808 P a885359a82bd4515913f7b273591f851 [term 1 LEADER]: Becoming Leader. State: Replica: a885359a82bd4515913f7b273591f851, State: Running, Role: LEADER
I20240502 01:52:49.531883 11446 consensus_queue.cc:237] T 673fc226d6b441219d1153741b914808 P a885359a82bd4515913f7b273591f851 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } }
I20240502 01:52:49.534833  9967 catalog_manager.cc:5484] T 673fc226d6b441219d1153741b914808 P a885359a82bd4515913f7b273591f851 reported cstate change: term changed from 0 to 1, leader changed from <none> to a885359a82bd4515913f7b273591f851 (127.9.176.193). New cstate: current_term: 1 leader_uuid: "a885359a82bd4515913f7b273591f851" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } health_report { overall_health: UNKNOWN } } }
I20240502 01:52:49.646031 11593 raft_consensus.cc:482] T 3a4f610cdc9d4dd48fd44924e406d929 P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:49.646212 11593 raft_consensus.cc:504] T 3a4f610cdc9d4dd48fd44924e406d929 P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } } peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } }
I20240502 01:52:49.646925 11593 leader_election.cc:288] T 3a4f610cdc9d4dd48fd44924e406d929 P fd95bcd4cdcb4cc5966050ffe2cffdff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b7a47245f294ce9a264fcc8f6351c0f (127.9.176.194:43645), 34da849acb2045a9a2ef03933592dd08 (127.9.176.196:44521)
I20240502 01:52:49.647554 10512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a4f610cdc9d4dd48fd44924e406d929" candidate_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34da849acb2045a9a2ef03933592dd08" is_pre_election: true
I20240502 01:52:49.647629 10246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a4f610cdc9d4dd48fd44924e406d929" candidate_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" is_pre_election: true
I20240502 01:52:49.647853 10246 raft_consensus.cc:2454] T 3a4f610cdc9d4dd48fd44924e406d929 P 7b7a47245f294ce9a264fcc8f6351c0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd95bcd4cdcb4cc5966050ffe2cffdff in term 0.
I20240502 01:52:49.647878 10512 raft_consensus.cc:2454] T 3a4f610cdc9d4dd48fd44924e406d929 P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd95bcd4cdcb4cc5966050ffe2cffdff in term 0.
I20240502 01:52:49.648401 10711 leader_election.cc:302] T 3a4f610cdc9d4dd48fd44924e406d929 P fd95bcd4cdcb4cc5966050ffe2cffdff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34da849acb2045a9a2ef03933592dd08, fd95bcd4cdcb4cc5966050ffe2cffdff; no voters: 
I20240502 01:52:49.648741 11593 raft_consensus.cc:2781] T 3a4f610cdc9d4dd48fd44924e406d929 P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:49.648883 11593 raft_consensus.cc:482] T 3a4f610cdc9d4dd48fd44924e406d929 P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:49.648977 11593 raft_consensus.cc:3037] T 3a4f610cdc9d4dd48fd44924e406d929 P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:49.650943 11593 raft_consensus.cc:504] T 3a4f610cdc9d4dd48fd44924e406d929 P fd95bcd4cdcb4cc5966050ffe2cffdff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } } peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } }
I20240502 01:52:49.651630 11593 leader_election.cc:288] T 3a4f610cdc9d4dd48fd44924e406d929 P fd95bcd4cdcb4cc5966050ffe2cffdff [CANDIDATE]: Term 1 election: Requested vote from peers 7b7a47245f294ce9a264fcc8f6351c0f (127.9.176.194:43645), 34da849acb2045a9a2ef03933592dd08 (127.9.176.196:44521)
I20240502 01:52:49.652357 10512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a4f610cdc9d4dd48fd44924e406d929" candidate_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34da849acb2045a9a2ef03933592dd08"
I20240502 01:52:49.652349 10246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a4f610cdc9d4dd48fd44924e406d929" candidate_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b7a47245f294ce9a264fcc8f6351c0f"
I20240502 01:52:49.652612 10246 raft_consensus.cc:3037] T 3a4f610cdc9d4dd48fd44924e406d929 P 7b7a47245f294ce9a264fcc8f6351c0f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:49.652634 10512 raft_consensus.cc:3037] T 3a4f610cdc9d4dd48fd44924e406d929 P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:49.655721 10246 raft_consensus.cc:2454] T 3a4f610cdc9d4dd48fd44924e406d929 P 7b7a47245f294ce9a264fcc8f6351c0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd95bcd4cdcb4cc5966050ffe2cffdff in term 1.
I20240502 01:52:49.655707 10512 raft_consensus.cc:2454] T 3a4f610cdc9d4dd48fd44924e406d929 P 34da849acb2045a9a2ef03933592dd08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd95bcd4cdcb4cc5966050ffe2cffdff in term 1.
I20240502 01:52:49.656342 10711 leader_election.cc:302] T 3a4f610cdc9d4dd48fd44924e406d929 P fd95bcd4cdcb4cc5966050ffe2cffdff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34da849acb2045a9a2ef03933592dd08, fd95bcd4cdcb4cc5966050ffe2cffdff; no voters: 
I20240502 01:52:49.656736 11593 raft_consensus.cc:2781] T 3a4f610cdc9d4dd48fd44924e406d929 P fd95bcd4cdcb4cc5966050ffe2cffdff [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:49.656904 11593 raft_consensus.cc:686] T 3a4f610cdc9d4dd48fd44924e406d929 P fd95bcd4cdcb4cc5966050ffe2cffdff [term 1 LEADER]: Becoming Leader. State: Replica: fd95bcd4cdcb4cc5966050ffe2cffdff, State: Running, Role: LEADER
I20240502 01:52:49.657243 11593 consensus_queue.cc:237] T 3a4f610cdc9d4dd48fd44924e406d929 P fd95bcd4cdcb4cc5966050ffe2cffdff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } } peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } }
I20240502 01:52:49.661837  9967 catalog_manager.cc:5484] T 3a4f610cdc9d4dd48fd44924e406d929 P fd95bcd4cdcb4cc5966050ffe2cffdff reported cstate change: term changed from 0 to 1, leader changed from <none> to fd95bcd4cdcb4cc5966050ffe2cffdff (127.9.176.198). New cstate: current_term: 1 leader_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } health_report { overall_health: UNKNOWN } } }
I20240502 01:52:49.967043 11574 consensus_queue.cc:1035] T c9f395cabd05425484aad0f82b0a3ca9 P a8c4a518929d4bb3a88bdfb1649493bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:49.972832 11574 consensus_queue.cc:1035] T c9f395cabd05425484aad0f82b0a3ca9 P a8c4a518929d4bb3a88bdfb1649493bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:52:50.039651 11636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:50.040012 11636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:50.048892 11636 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 01:52:50.053418 11446 consensus_queue.cc:1035] T 673fc226d6b441219d1153741b914808 P a885359a82bd4515913f7b273591f851 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:50.067183 11647 consensus_queue.cc:1035] T 673fc226d6b441219d1153741b914808 P a885359a82bd4515913f7b273591f851 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_8'
I20240502 01:52:50.153759 11477 consensus_queue.cc:1035] T 3a4f610cdc9d4dd48fd44924e406d929 P fd95bcd4cdcb4cc5966050ffe2cffdff [LEADER]: Connected to new peer: Peer: permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:50.189177 11593 consensus_queue.cc:1035] T 3a4f610cdc9d4dd48fd44924e406d929 P fd95bcd4cdcb4cc5966050ffe2cffdff [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:52:50.373571 10443 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.176.194:43645, user_credentials={real_user=slave}} blocked reactor thread for 53955.6us

INSERT report
    rows total: 2000
    time total: 325.756 ms
  time per row: 0.162878 ms
-----------------------------------------------------
Suppressions used:
  count      bytes template
      2        596 libcrypto.so.1.1
-----------------------------------------------------

I20240502 01:52:51.553871  9960 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56524:
name: "rebalance_test_table_9"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 01:52:51.565191 10759 tablet_service.cc:1467] Processing CreateTablet for tablet e185fe1a710b47aca0df91cb77bf60e0 (DEFAULT_TABLE table=rebalance_test_table_9 [id=ffb7b6400eaa4f8284a345c895252a82]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:51.565770 10759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e185fe1a710b47aca0df91cb77bf60e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:51.565701 10225 tablet_service.cc:1467] Processing CreateTablet for tablet e185fe1a710b47aca0df91cb77bf60e0 (DEFAULT_TABLE table=rebalance_test_table_9 [id=ffb7b6400eaa4f8284a345c895252a82]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:51.565820 10492 tablet_service.cc:1467] Processing CreateTablet for tablet eaefd6b8be0947da8d8cb87644f91c70 (DEFAULT_TABLE table=rebalance_test_table_9 [id=ffb7b6400eaa4f8284a345c895252a82]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:51.566236 10225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e185fe1a710b47aca0df91cb77bf60e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:51.566256 10492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eaefd6b8be0947da8d8cb87644f91c70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:51.566565 10091 tablet_service.cc:1467] Processing CreateTablet for tablet dea628b438e74180a688e5dca1f439ab (DEFAULT_TABLE table=rebalance_test_table_9 [id=ffb7b6400eaa4f8284a345c895252a82]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:51.566995 10091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dea628b438e74180a688e5dca1f439ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:51.572062 10491 tablet_service.cc:1467] Processing CreateTablet for tablet e185fe1a710b47aca0df91cb77bf60e0 (DEFAULT_TABLE table=rebalance_test_table_9 [id=ffb7b6400eaa4f8284a345c895252a82]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:51.572595 10491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e185fe1a710b47aca0df91cb77bf60e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:51.573004 10758 tablet_service.cc:1467] Processing CreateTablet for tablet eaefd6b8be0947da8d8cb87644f91c70 (DEFAULT_TABLE table=rebalance_test_table_9 [id=ffb7b6400eaa4f8284a345c895252a82]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:51.573487 10758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eaefd6b8be0947da8d8cb87644f91c70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:51.573669 10224 tablet_service.cc:1467] Processing CreateTablet for tablet eaefd6b8be0947da8d8cb87644f91c70 (DEFAULT_TABLE table=rebalance_test_table_9 [id=ffb7b6400eaa4f8284a345c895252a82]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:51.574090 10224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eaefd6b8be0947da8d8cb87644f91c70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:51.574306 10623 tablet_service.cc:1467] Processing CreateTablet for tablet dea628b438e74180a688e5dca1f439ab (DEFAULT_TABLE table=rebalance_test_table_9 [id=ffb7b6400eaa4f8284a345c895252a82]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:51.574730 10623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dea628b438e74180a688e5dca1f439ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:51.574700 10358 tablet_service.cc:1467] Processing CreateTablet for tablet dea628b438e74180a688e5dca1f439ab (DEFAULT_TABLE table=rebalance_test_table_9 [id=ffb7b6400eaa4f8284a345c895252a82]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:51.575122 10358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dea628b438e74180a688e5dca1f439ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:51.585649 11712 tablet_bootstrap.cc:492] T dea628b438e74180a688e5dca1f439ab P a8c4a518929d4bb3a88bdfb1649493bd: Bootstrap starting.
I20240502 01:52:51.588508 11712 tablet_bootstrap.cc:654] T dea628b438e74180a688e5dca1f439ab P a8c4a518929d4bb3a88bdfb1649493bd: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:51.589270 11710 tablet_bootstrap.cc:492] T e185fe1a710b47aca0df91cb77bf60e0 P 7b7a47245f294ce9a264fcc8f6351c0f: Bootstrap starting.
I20240502 01:52:51.593394 11710 tablet_bootstrap.cc:654] T e185fe1a710b47aca0df91cb77bf60e0 P 7b7a47245f294ce9a264fcc8f6351c0f: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:51.600320 11711 tablet_bootstrap.cc:492] T eaefd6b8be0947da8d8cb87644f91c70 P 34da849acb2045a9a2ef03933592dd08: Bootstrap starting.
I20240502 01:52:51.604269 11714 tablet_bootstrap.cc:492] T dea628b438e74180a688e5dca1f439ab P ede1dd598e3541138a19a3ea5c052bc9: Bootstrap starting.
I20240502 01:52:51.604329 11712 tablet_bootstrap.cc:492] T dea628b438e74180a688e5dca1f439ab P a8c4a518929d4bb3a88bdfb1649493bd: No bootstrap required, opened a new log
I20240502 01:52:51.604511 11712 ts_tablet_manager.cc:1367] T dea628b438e74180a688e5dca1f439ab P a8c4a518929d4bb3a88bdfb1649493bd: Time spent bootstrapping tablet: real 0.019s	user 0.009s	sys 0.000s
I20240502 01:52:51.605481 11712 raft_consensus.cc:348] T dea628b438e74180a688e5dca1f439ab P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } }
I20240502 01:52:51.605789 11712 raft_consensus.cc:374] T dea628b438e74180a688e5dca1f439ab P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:51.605939 11712 raft_consensus.cc:729] T dea628b438e74180a688e5dca1f439ab P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8c4a518929d4bb3a88bdfb1649493bd, State: Initialized, Role: FOLLOWER
I20240502 01:52:51.606235 11712 consensus_queue.cc:260] T dea628b438e74180a688e5dca1f439ab P a8c4a518929d4bb3a88bdfb1649493bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } }
I20240502 01:52:51.607334 11712 ts_tablet_manager.cc:1398] T dea628b438e74180a688e5dca1f439ab P a8c4a518929d4bb3a88bdfb1649493bd: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 01:52:51.608577 11713 tablet_bootstrap.cc:492] T dea628b438e74180a688e5dca1f439ab P a885359a82bd4515913f7b273591f851: Bootstrap starting.
I20240502 01:52:51.610407 11713 tablet_bootstrap.cc:654] T dea628b438e74180a688e5dca1f439ab P a885359a82bd4515913f7b273591f851: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:51.612149 11711 tablet_bootstrap.cc:654] T eaefd6b8be0947da8d8cb87644f91c70 P 34da849acb2045a9a2ef03933592dd08: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:51.612596 11714 tablet_bootstrap.cc:654] T dea628b438e74180a688e5dca1f439ab P ede1dd598e3541138a19a3ea5c052bc9: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:51.614166 11709 tablet_bootstrap.cc:492] T eaefd6b8be0947da8d8cb87644f91c70 P fd95bcd4cdcb4cc5966050ffe2cffdff: Bootstrap starting.
I20240502 01:52:51.615098 11711 tablet_bootstrap.cc:492] T eaefd6b8be0947da8d8cb87644f91c70 P 34da849acb2045a9a2ef03933592dd08: No bootstrap required, opened a new log
I20240502 01:52:51.615330 11711 ts_tablet_manager.cc:1367] T eaefd6b8be0947da8d8cb87644f91c70 P 34da849acb2045a9a2ef03933592dd08: Time spent bootstrapping tablet: real 0.015s	user 0.005s	sys 0.000s
I20240502 01:52:51.615849 11709 tablet_bootstrap.cc:654] T eaefd6b8be0947da8d8cb87644f91c70 P fd95bcd4cdcb4cc5966050ffe2cffdff: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:51.616256 11711 raft_consensus.cc:348] T eaefd6b8be0947da8d8cb87644f91c70 P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } } peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } }
I20240502 01:52:51.616561 11711 raft_consensus.cc:374] T eaefd6b8be0947da8d8cb87644f91c70 P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:51.616680 11711 raft_consensus.cc:729] T eaefd6b8be0947da8d8cb87644f91c70 P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34da849acb2045a9a2ef03933592dd08, State: Initialized, Role: FOLLOWER
I20240502 01:52:51.616986 11714 tablet_bootstrap.cc:492] T dea628b438e74180a688e5dca1f439ab P ede1dd598e3541138a19a3ea5c052bc9: No bootstrap required, opened a new log
I20240502 01:52:51.616982 11711 consensus_queue.cc:260] T eaefd6b8be0947da8d8cb87644f91c70 P 34da849acb2045a9a2ef03933592dd08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } } peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } }
I20240502 01:52:51.617169 11714 ts_tablet_manager.cc:1367] T dea628b438e74180a688e5dca1f439ab P ede1dd598e3541138a19a3ea5c052bc9: Time spent bootstrapping tablet: real 0.013s	user 0.004s	sys 0.000s
I20240502 01:52:51.617695 11711 ts_tablet_manager.cc:1398] T eaefd6b8be0947da8d8cb87644f91c70 P 34da849acb2045a9a2ef03933592dd08: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240502 01:52:51.618023 11714 raft_consensus.cc:348] T dea628b438e74180a688e5dca1f439ab P ede1dd598e3541138a19a3ea5c052bc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } }
I20240502 01:52:51.618247 11714 raft_consensus.cc:374] T dea628b438e74180a688e5dca1f439ab P ede1dd598e3541138a19a3ea5c052bc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:51.618396 11714 raft_consensus.cc:729] T dea628b438e74180a688e5dca1f439ab P ede1dd598e3541138a19a3ea5c052bc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ede1dd598e3541138a19a3ea5c052bc9, State: Initialized, Role: FOLLOWER
I20240502 01:52:51.618675 11714 consensus_queue.cc:260] T dea628b438e74180a688e5dca1f439ab P ede1dd598e3541138a19a3ea5c052bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } }
I20240502 01:52:51.619081 11711 tablet_bootstrap.cc:492] T e185fe1a710b47aca0df91cb77bf60e0 P 34da849acb2045a9a2ef03933592dd08: Bootstrap starting.
I20240502 01:52:51.619663 11713 tablet_bootstrap.cc:492] T dea628b438e74180a688e5dca1f439ab P a885359a82bd4515913f7b273591f851: No bootstrap required, opened a new log
I20240502 01:52:51.619908 11713 ts_tablet_manager.cc:1367] T dea628b438e74180a688e5dca1f439ab P a885359a82bd4515913f7b273591f851: Time spent bootstrapping tablet: real 0.011s	user 0.004s	sys 0.000s
I20240502 01:52:51.620158 11709 tablet_bootstrap.cc:492] T eaefd6b8be0947da8d8cb87644f91c70 P fd95bcd4cdcb4cc5966050ffe2cffdff: No bootstrap required, opened a new log
I20240502 01:52:51.620188 11710 tablet_bootstrap.cc:492] T e185fe1a710b47aca0df91cb77bf60e0 P 7b7a47245f294ce9a264fcc8f6351c0f: No bootstrap required, opened a new log
I20240502 01:52:51.620381 11710 ts_tablet_manager.cc:1367] T e185fe1a710b47aca0df91cb77bf60e0 P 7b7a47245f294ce9a264fcc8f6351c0f: Time spent bootstrapping tablet: real 0.031s	user 0.006s	sys 0.000s
I20240502 01:52:51.620415 11709 ts_tablet_manager.cc:1367] T eaefd6b8be0947da8d8cb87644f91c70 P fd95bcd4cdcb4cc5966050ffe2cffdff: Time spent bootstrapping tablet: real 0.006s	user 0.004s	sys 0.000s
I20240502 01:52:51.620877 11713 raft_consensus.cc:348] T dea628b438e74180a688e5dca1f439ab P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } }
I20240502 01:52:51.621177 11713 raft_consensus.cc:374] T dea628b438e74180a688e5dca1f439ab P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:51.621213 11710 raft_consensus.cc:348] T e185fe1a710b47aca0df91cb77bf60e0 P 7b7a47245f294ce9a264fcc8f6351c0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } }
I20240502 01:52:51.621335 11713 raft_consensus.cc:729] T dea628b438e74180a688e5dca1f439ab P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a885359a82bd4515913f7b273591f851, State: Initialized, Role: FOLLOWER
I20240502 01:52:51.621433 11710 raft_consensus.cc:374] T e185fe1a710b47aca0df91cb77bf60e0 P 7b7a47245f294ce9a264fcc8f6351c0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:51.621536 11710 raft_consensus.cc:729] T e185fe1a710b47aca0df91cb77bf60e0 P 7b7a47245f294ce9a264fcc8f6351c0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b7a47245f294ce9a264fcc8f6351c0f, State: Initialized, Role: FOLLOWER
I20240502 01:52:51.621716 11713 consensus_queue.cc:260] T dea628b438e74180a688e5dca1f439ab P a885359a82bd4515913f7b273591f851 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } }
I20240502 01:52:51.621796 11710 consensus_queue.cc:260] T e185fe1a710b47aca0df91cb77bf60e0 P 7b7a47245f294ce9a264fcc8f6351c0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } }
I20240502 01:52:51.622327 11710 ts_tablet_manager.cc:1398] T e185fe1a710b47aca0df91cb77bf60e0 P 7b7a47245f294ce9a264fcc8f6351c0f: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 01:52:51.622339 11713 ts_tablet_manager.cc:1398] T dea628b438e74180a688e5dca1f439ab P a885359a82bd4515913f7b273591f851: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240502 01:52:51.622653 11710 tablet_bootstrap.cc:492] T eaefd6b8be0947da8d8cb87644f91c70 P 7b7a47245f294ce9a264fcc8f6351c0f: Bootstrap starting.
I20240502 01:52:51.621343 11709 raft_consensus.cc:348] T eaefd6b8be0947da8d8cb87644f91c70 P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } } peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } }
I20240502 01:52:51.624138 11709 raft_consensus.cc:374] T eaefd6b8be0947da8d8cb87644f91c70 P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:51.624305 11709 raft_consensus.cc:729] T eaefd6b8be0947da8d8cb87644f91c70 P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd95bcd4cdcb4cc5966050ffe2cffdff, State: Initialized, Role: FOLLOWER
I20240502 01:52:51.624608 11709 consensus_queue.cc:260] T eaefd6b8be0947da8d8cb87644f91c70 P fd95bcd4cdcb4cc5966050ffe2cffdff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } } peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } }
I20240502 01:52:51.625321 11709 ts_tablet_manager.cc:1398] T eaefd6b8be0947da8d8cb87644f91c70 P fd95bcd4cdcb4cc5966050ffe2cffdff: Time spent starting tablet: real 0.005s	user 0.002s	sys 0.000s
I20240502 01:52:51.626467 11709 tablet_bootstrap.cc:492] T e185fe1a710b47aca0df91cb77bf60e0 P fd95bcd4cdcb4cc5966050ffe2cffdff: Bootstrap starting.
I20240502 01:52:51.626724 11710 tablet_bootstrap.cc:654] T eaefd6b8be0947da8d8cb87644f91c70 P 7b7a47245f294ce9a264fcc8f6351c0f: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:51.628293 11709 tablet_bootstrap.cc:654] T e185fe1a710b47aca0df91cb77bf60e0 P fd95bcd4cdcb4cc5966050ffe2cffdff: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:51.632977 11714 ts_tablet_manager.cc:1398] T dea628b438e74180a688e5dca1f439ab P ede1dd598e3541138a19a3ea5c052bc9: Time spent starting tablet: real 0.016s	user 0.003s	sys 0.000s
I20240502 01:52:51.639143 11711 tablet_bootstrap.cc:654] T e185fe1a710b47aca0df91cb77bf60e0 P 34da849acb2045a9a2ef03933592dd08: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:51.643832 11710 tablet_bootstrap.cc:492] T eaefd6b8be0947da8d8cb87644f91c70 P 7b7a47245f294ce9a264fcc8f6351c0f: No bootstrap required, opened a new log
I20240502 01:52:51.644251 11710 ts_tablet_manager.cc:1367] T eaefd6b8be0947da8d8cb87644f91c70 P 7b7a47245f294ce9a264fcc8f6351c0f: Time spent bootstrapping tablet: real 0.022s	user 0.001s	sys 0.011s
I20240502 01:52:51.645576 11710 raft_consensus.cc:348] T eaefd6b8be0947da8d8cb87644f91c70 P 7b7a47245f294ce9a264fcc8f6351c0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } } peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } }
I20240502 01:52:51.646173 11710 raft_consensus.cc:374] T eaefd6b8be0947da8d8cb87644f91c70 P 7b7a47245f294ce9a264fcc8f6351c0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:51.646342 11710 raft_consensus.cc:729] T eaefd6b8be0947da8d8cb87644f91c70 P 7b7a47245f294ce9a264fcc8f6351c0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b7a47245f294ce9a264fcc8f6351c0f, State: Initialized, Role: FOLLOWER
I20240502 01:52:51.647061 11710 consensus_queue.cc:260] T eaefd6b8be0947da8d8cb87644f91c70 P 7b7a47245f294ce9a264fcc8f6351c0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } } peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } }
I20240502 01:52:51.650133 11711 tablet_bootstrap.cc:492] T e185fe1a710b47aca0df91cb77bf60e0 P 34da849acb2045a9a2ef03933592dd08: No bootstrap required, opened a new log
I20240502 01:52:51.650430 11711 ts_tablet_manager.cc:1367] T e185fe1a710b47aca0df91cb77bf60e0 P 34da849acb2045a9a2ef03933592dd08: Time spent bootstrapping tablet: real 0.031s	user 0.000s	sys 0.006s
I20240502 01:52:51.650619 11710 ts_tablet_manager.cc:1398] T eaefd6b8be0947da8d8cb87644f91c70 P 7b7a47245f294ce9a264fcc8f6351c0f: Time spent starting tablet: real 0.006s	user 0.000s	sys 0.003s
I20240502 01:52:51.651955 11711 raft_consensus.cc:348] T e185fe1a710b47aca0df91cb77bf60e0 P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } }
I20240502 01:52:51.652380 11711 raft_consensus.cc:374] T e185fe1a710b47aca0df91cb77bf60e0 P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:51.652541 11711 raft_consensus.cc:729] T e185fe1a710b47aca0df91cb77bf60e0 P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34da849acb2045a9a2ef03933592dd08, State: Initialized, Role: FOLLOWER
I20240502 01:52:51.652874 11711 consensus_queue.cc:260] T e185fe1a710b47aca0df91cb77bf60e0 P 34da849acb2045a9a2ef03933592dd08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } }
I20240502 01:52:51.653903 11711 ts_tablet_manager.cc:1398] T e185fe1a710b47aca0df91cb77bf60e0 P 34da849acb2045a9a2ef03933592dd08: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.002s
I20240502 01:52:51.655503 11709 tablet_bootstrap.cc:492] T e185fe1a710b47aca0df91cb77bf60e0 P fd95bcd4cdcb4cc5966050ffe2cffdff: No bootstrap required, opened a new log
I20240502 01:52:51.655805 11709 ts_tablet_manager.cc:1367] T e185fe1a710b47aca0df91cb77bf60e0 P fd95bcd4cdcb4cc5966050ffe2cffdff: Time spent bootstrapping tablet: real 0.029s	user 0.007s	sys 0.020s
I20240502 01:52:51.656666 11709 raft_consensus.cc:348] T e185fe1a710b47aca0df91cb77bf60e0 P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } }
I20240502 01:52:51.656934 11709 raft_consensus.cc:374] T e185fe1a710b47aca0df91cb77bf60e0 P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:51.657021 11709 raft_consensus.cc:729] T e185fe1a710b47aca0df91cb77bf60e0 P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd95bcd4cdcb4cc5966050ffe2cffdff, State: Initialized, Role: FOLLOWER
I20240502 01:52:51.657287 11709 consensus_queue.cc:260] T e185fe1a710b47aca0df91cb77bf60e0 P fd95bcd4cdcb4cc5966050ffe2cffdff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } }
I20240502 01:52:51.657902 11709 ts_tablet_manager.cc:1398] T e185fe1a710b47aca0df91cb77bf60e0 P fd95bcd4cdcb4cc5966050ffe2cffdff: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240502 01:52:51.682644 11565 raft_consensus.cc:482] T e185fe1a710b47aca0df91cb77bf60e0 P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:51.682931 11565 raft_consensus.cc:504] T e185fe1a710b47aca0df91cb77bf60e0 P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } }
I20240502 01:52:51.683897 11565 leader_election.cc:288] T e185fe1a710b47aca0df91cb77bf60e0 P 34da849acb2045a9a2ef03933592dd08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fd95bcd4cdcb4cc5966050ffe2cffdff (127.9.176.198:33039), 7b7a47245f294ce9a264fcc8f6351c0f (127.9.176.194:43645)
I20240502 01:52:51.684587 10246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e185fe1a710b47aca0df91cb77bf60e0" candidate_uuid: "34da849acb2045a9a2ef03933592dd08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" is_pre_election: true
I20240502 01:52:51.684774 10779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e185fe1a710b47aca0df91cb77bf60e0" candidate_uuid: "34da849acb2045a9a2ef03933592dd08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" is_pre_election: true
I20240502 01:52:51.684935 10246 raft_consensus.cc:2454] T e185fe1a710b47aca0df91cb77bf60e0 P 7b7a47245f294ce9a264fcc8f6351c0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34da849acb2045a9a2ef03933592dd08 in term 0.
I20240502 01:52:51.685123 10779 raft_consensus.cc:2454] T e185fe1a710b47aca0df91cb77bf60e0 P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34da849acb2045a9a2ef03933592dd08 in term 0.
I20240502 01:52:51.685483 10443 leader_election.cc:302] T e185fe1a710b47aca0df91cb77bf60e0 P 34da849acb2045a9a2ef03933592dd08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34da849acb2045a9a2ef03933592dd08, 7b7a47245f294ce9a264fcc8f6351c0f; no voters: 
I20240502 01:52:51.685855 11565 raft_consensus.cc:2781] T e185fe1a710b47aca0df91cb77bf60e0 P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:51.685957 11565 raft_consensus.cc:482] T e185fe1a710b47aca0df91cb77bf60e0 P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:51.686043 11565 raft_consensus.cc:3037] T e185fe1a710b47aca0df91cb77bf60e0 P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:51.749294 11565 raft_consensus.cc:504] T e185fe1a710b47aca0df91cb77bf60e0 P 34da849acb2045a9a2ef03933592dd08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } }
I20240502 01:52:51.750196 11565 leader_election.cc:288] T e185fe1a710b47aca0df91cb77bf60e0 P 34da849acb2045a9a2ef03933592dd08 [CANDIDATE]: Term 1 election: Requested vote from peers fd95bcd4cdcb4cc5966050ffe2cffdff (127.9.176.198:33039), 7b7a47245f294ce9a264fcc8f6351c0f (127.9.176.194:43645)
I20240502 01:52:51.750825 10779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e185fe1a710b47aca0df91cb77bf60e0" candidate_uuid: "34da849acb2045a9a2ef03933592dd08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff"
I20240502 01:52:51.750952 10246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e185fe1a710b47aca0df91cb77bf60e0" candidate_uuid: "34da849acb2045a9a2ef03933592dd08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b7a47245f294ce9a264fcc8f6351c0f"
I20240502 01:52:51.751156 10779 raft_consensus.cc:3037] T e185fe1a710b47aca0df91cb77bf60e0 P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:51.751219 10246 raft_consensus.cc:3037] T e185fe1a710b47aca0df91cb77bf60e0 P 7b7a47245f294ce9a264fcc8f6351c0f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:51.754402 10779 raft_consensus.cc:2454] T e185fe1a710b47aca0df91cb77bf60e0 P fd95bcd4cdcb4cc5966050ffe2cffdff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34da849acb2045a9a2ef03933592dd08 in term 1.
I20240502 01:52:51.754391 10246 raft_consensus.cc:2454] T e185fe1a710b47aca0df91cb77bf60e0 P 7b7a47245f294ce9a264fcc8f6351c0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34da849acb2045a9a2ef03933592dd08 in term 1.
I20240502 01:52:51.755050 10443 leader_election.cc:302] T e185fe1a710b47aca0df91cb77bf60e0 P 34da849acb2045a9a2ef03933592dd08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34da849acb2045a9a2ef03933592dd08, 7b7a47245f294ce9a264fcc8f6351c0f; no voters: 
I20240502 01:52:51.755426 11565 raft_consensus.cc:2781] T e185fe1a710b47aca0df91cb77bf60e0 P 34da849acb2045a9a2ef03933592dd08 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:51.755723 11565 raft_consensus.cc:686] T e185fe1a710b47aca0df91cb77bf60e0 P 34da849acb2045a9a2ef03933592dd08 [term 1 LEADER]: Becoming Leader. State: Replica: 34da849acb2045a9a2ef03933592dd08, State: Running, Role: LEADER
I20240502 01:52:51.756065 11565 consensus_queue.cc:237] T e185fe1a710b47aca0df91cb77bf60e0 P 34da849acb2045a9a2ef03933592dd08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } }
I20240502 01:52:51.760464  9960 catalog_manager.cc:5484] T e185fe1a710b47aca0df91cb77bf60e0 P 34da849acb2045a9a2ef03933592dd08 reported cstate change: term changed from 0 to 1, leader changed from <none> to 34da849acb2045a9a2ef03933592dd08 (127.9.176.196). New cstate: current_term: 1 leader_uuid: "34da849acb2045a9a2ef03933592dd08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } health_report { overall_health: UNKNOWN } } }
I20240502 01:52:51.780134 11704 raft_consensus.cc:482] T dea628b438e74180a688e5dca1f439ab P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:51.780400 11704 raft_consensus.cc:504] T dea628b438e74180a688e5dca1f439ab P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } }
I20240502 01:52:51.781172 11704 leader_election.cc:288] T dea628b438e74180a688e5dca1f439ab P a885359a82bd4515913f7b273591f851 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a8c4a518929d4bb3a88bdfb1649493bd (127.9.176.197:44061), ede1dd598e3541138a19a3ea5c052bc9 (127.9.176.195:38701)
I20240502 01:52:51.781868 10377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dea628b438e74180a688e5dca1f439ab" candidate_uuid: "a885359a82bd4515913f7b273591f851" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ede1dd598e3541138a19a3ea5c052bc9" is_pre_election: true
I20240502 01:52:51.781869 10644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dea628b438e74180a688e5dca1f439ab" candidate_uuid: "a885359a82bd4515913f7b273591f851" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" is_pre_election: true
I20240502 01:52:51.782135 10377 raft_consensus.cc:2454] T dea628b438e74180a688e5dca1f439ab P ede1dd598e3541138a19a3ea5c052bc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a885359a82bd4515913f7b273591f851 in term 0.
I20240502 01:52:51.782151 10644 raft_consensus.cc:2454] T dea628b438e74180a688e5dca1f439ab P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a885359a82bd4515913f7b273591f851 in term 0.
I20240502 01:52:51.782770 10045 leader_election.cc:302] T dea628b438e74180a688e5dca1f439ab P a885359a82bd4515913f7b273591f851 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: a885359a82bd4515913f7b273591f851, a8c4a518929d4bb3a88bdfb1649493bd, ede1dd598e3541138a19a3ea5c052bc9; no voters: 
I20240502 01:52:51.783123 11704 raft_consensus.cc:2781] T dea628b438e74180a688e5dca1f439ab P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:51.783257 11704 raft_consensus.cc:482] T dea628b438e74180a688e5dca1f439ab P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:51.783389 11704 raft_consensus.cc:3037] T dea628b438e74180a688e5dca1f439ab P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:51.785567 11704 raft_consensus.cc:504] T dea628b438e74180a688e5dca1f439ab P a885359a82bd4515913f7b273591f851 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } }
I20240502 01:52:51.786286 11704 leader_election.cc:288] T dea628b438e74180a688e5dca1f439ab P a885359a82bd4515913f7b273591f851 [CANDIDATE]: Term 1 election: Requested vote from peers a8c4a518929d4bb3a88bdfb1649493bd (127.9.176.197:44061), ede1dd598e3541138a19a3ea5c052bc9 (127.9.176.195:38701)
I20240502 01:52:51.786824 10644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dea628b438e74180a688e5dca1f439ab" candidate_uuid: "a885359a82bd4515913f7b273591f851" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8c4a518929d4bb3a88bdfb1649493bd"
I20240502 01:52:51.786919 10377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dea628b438e74180a688e5dca1f439ab" candidate_uuid: "a885359a82bd4515913f7b273591f851" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ede1dd598e3541138a19a3ea5c052bc9"
I20240502 01:52:51.787082 10644 raft_consensus.cc:3037] T dea628b438e74180a688e5dca1f439ab P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:51.787146 10377 raft_consensus.cc:3037] T dea628b438e74180a688e5dca1f439ab P ede1dd598e3541138a19a3ea5c052bc9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:51.790267 10644 raft_consensus.cc:2454] T dea628b438e74180a688e5dca1f439ab P a8c4a518929d4bb3a88bdfb1649493bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a885359a82bd4515913f7b273591f851 in term 1.
I20240502 01:52:51.790267 10377 raft_consensus.cc:2454] T dea628b438e74180a688e5dca1f439ab P ede1dd598e3541138a19a3ea5c052bc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a885359a82bd4515913f7b273591f851 in term 1.
I20240502 01:52:51.790911 10045 leader_election.cc:302] T dea628b438e74180a688e5dca1f439ab P a885359a82bd4515913f7b273591f851 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: a885359a82bd4515913f7b273591f851, a8c4a518929d4bb3a88bdfb1649493bd, ede1dd598e3541138a19a3ea5c052bc9; no voters: 
I20240502 01:52:51.791389 11704 raft_consensus.cc:2781] T dea628b438e74180a688e5dca1f439ab P a885359a82bd4515913f7b273591f851 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:51.791615 11704 raft_consensus.cc:686] T dea628b438e74180a688e5dca1f439ab P a885359a82bd4515913f7b273591f851 [term 1 LEADER]: Becoming Leader. State: Replica: a885359a82bd4515913f7b273591f851, State: Running, Role: LEADER
I20240502 01:52:51.791951 11704 consensus_queue.cc:237] T dea628b438e74180a688e5dca1f439ab P a885359a82bd4515913f7b273591f851 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } }
I20240502 01:52:51.796054  9960 catalog_manager.cc:5484] T dea628b438e74180a688e5dca1f439ab P a885359a82bd4515913f7b273591f851 reported cstate change: term changed from 0 to 1, leader changed from <none> to a885359a82bd4515913f7b273591f851 (127.9.176.193). New cstate: current_term: 1 leader_uuid: "a885359a82bd4515913f7b273591f851" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } health_report { overall_health: UNKNOWN } } }
I20240502 01:52:51.806659 11727 raft_consensus.cc:482] T eaefd6b8be0947da8d8cb87644f91c70 P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:51.806885 11727 raft_consensus.cc:504] T eaefd6b8be0947da8d8cb87644f91c70 P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } } peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } }
I20240502 01:52:51.807677 11727 leader_election.cc:288] T eaefd6b8be0947da8d8cb87644f91c70 P fd95bcd4cdcb4cc5966050ffe2cffdff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 34da849acb2045a9a2ef03933592dd08 (127.9.176.196:44521), 7b7a47245f294ce9a264fcc8f6351c0f (127.9.176.194:43645)
I20240502 01:52:51.808389 10511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaefd6b8be0947da8d8cb87644f91c70" candidate_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34da849acb2045a9a2ef03933592dd08" is_pre_election: true
I20240502 01:52:51.808390 10246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaefd6b8be0947da8d8cb87644f91c70" candidate_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" is_pre_election: true
I20240502 01:52:51.808658 10246 raft_consensus.cc:2454] T eaefd6b8be0947da8d8cb87644f91c70 P 7b7a47245f294ce9a264fcc8f6351c0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd95bcd4cdcb4cc5966050ffe2cffdff in term 0.
I20240502 01:52:51.808670 10511 raft_consensus.cc:2454] T eaefd6b8be0947da8d8cb87644f91c70 P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd95bcd4cdcb4cc5966050ffe2cffdff in term 0.
I20240502 01:52:51.809152 10711 leader_election.cc:302] T eaefd6b8be0947da8d8cb87644f91c70 P fd95bcd4cdcb4cc5966050ffe2cffdff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34da849acb2045a9a2ef03933592dd08, fd95bcd4cdcb4cc5966050ffe2cffdff; no voters: 
I20240502 01:52:51.809458 11727 raft_consensus.cc:2781] T eaefd6b8be0947da8d8cb87644f91c70 P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:51.809602 11727 raft_consensus.cc:482] T eaefd6b8be0947da8d8cb87644f91c70 P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:51.809693 11727 raft_consensus.cc:3037] T eaefd6b8be0947da8d8cb87644f91c70 P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:51.811776 11727 raft_consensus.cc:504] T eaefd6b8be0947da8d8cb87644f91c70 P fd95bcd4cdcb4cc5966050ffe2cffdff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } } peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } }
I20240502 01:52:51.812448 11727 leader_election.cc:288] T eaefd6b8be0947da8d8cb87644f91c70 P fd95bcd4cdcb4cc5966050ffe2cffdff [CANDIDATE]: Term 1 election: Requested vote from peers 34da849acb2045a9a2ef03933592dd08 (127.9.176.196:44521), 7b7a47245f294ce9a264fcc8f6351c0f (127.9.176.194:43645)
I20240502 01:52:51.813143 10246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaefd6b8be0947da8d8cb87644f91c70" candidate_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b7a47245f294ce9a264fcc8f6351c0f"
I20240502 01:52:51.813143 10511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaefd6b8be0947da8d8cb87644f91c70" candidate_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34da849acb2045a9a2ef03933592dd08"
I20240502 01:52:51.813387 10246 raft_consensus.cc:3037] T eaefd6b8be0947da8d8cb87644f91c70 P 7b7a47245f294ce9a264fcc8f6351c0f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:51.813387 10511 raft_consensus.cc:3037] T eaefd6b8be0947da8d8cb87644f91c70 P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:51.816462 10511 raft_consensus.cc:2454] T eaefd6b8be0947da8d8cb87644f91c70 P 34da849acb2045a9a2ef03933592dd08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd95bcd4cdcb4cc5966050ffe2cffdff in term 1.
I20240502 01:52:51.816725 10246 raft_consensus.cc:2454] T eaefd6b8be0947da8d8cb87644f91c70 P 7b7a47245f294ce9a264fcc8f6351c0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd95bcd4cdcb4cc5966050ffe2cffdff in term 1.
I20240502 01:52:51.817591 10711 leader_election.cc:302] T eaefd6b8be0947da8d8cb87644f91c70 P fd95bcd4cdcb4cc5966050ffe2cffdff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34da849acb2045a9a2ef03933592dd08, fd95bcd4cdcb4cc5966050ffe2cffdff; no voters: 
I20240502 01:52:51.817953 11727 raft_consensus.cc:2781] T eaefd6b8be0947da8d8cb87644f91c70 P fd95bcd4cdcb4cc5966050ffe2cffdff [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:51.818120 11727 raft_consensus.cc:686] T eaefd6b8be0947da8d8cb87644f91c70 P fd95bcd4cdcb4cc5966050ffe2cffdff [term 1 LEADER]: Becoming Leader. State: Replica: fd95bcd4cdcb4cc5966050ffe2cffdff, State: Running, Role: LEADER
I20240502 01:52:51.818470 11727 consensus_queue.cc:237] T eaefd6b8be0947da8d8cb87644f91c70 P fd95bcd4cdcb4cc5966050ffe2cffdff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } } peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } }
I20240502 01:52:51.822073  9960 catalog_manager.cc:5484] T eaefd6b8be0947da8d8cb87644f91c70 P fd95bcd4cdcb4cc5966050ffe2cffdff reported cstate change: term changed from 0 to 1, leader changed from <none> to fd95bcd4cdcb4cc5966050ffe2cffdff (127.9.176.198). New cstate: current_term: 1 leader_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } health_report { overall_health: UNKNOWN } } }
I20240502 01:52:52.162607 11565 consensus_queue.cc:1035] T e185fe1a710b47aca0df91cb77bf60e0 P 34da849acb2045a9a2ef03933592dd08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:52.169234 11565 consensus_queue.cc:1035] T e185fe1a710b47aca0df91cb77bf60e0 P 34da849acb2045a9a2ef03933592dd08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:52:52.234318 11730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:52.234794 11730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:52.244055 11730 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 01:52:52.256613 11728 consensus_queue.cc:1035] T eaefd6b8be0947da8d8cb87644f91c70 P fd95bcd4cdcb4cc5966050ffe2cffdff [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:52.264724 11704 consensus_queue.cc:1035] T dea628b438e74180a688e5dca1f439ab P a885359a82bd4515913f7b273591f851 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:52.265661 11693 consensus_queue.cc:1035] T eaefd6b8be0947da8d8cb87644f91c70 P fd95bcd4cdcb4cc5966050ffe2cffdff [LEADER]: Connected to new peer: Peer: permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:52.274525 11647 consensus_queue.cc:1035] T dea628b438e74180a688e5dca1f439ab P a885359a82bd4515913f7b273591f851 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_9'

INSERT report
    rows total: 2000
    time total: 276.338 ms
  time per row: 0.138169 ms
I20240502 01:52:52.706810  9961 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56524:
name: "rebalance_test_table_10"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 01:52:52.717305 10091 tablet_service.cc:1467] Processing CreateTablet for tablet d8ffb869041f41259d08443b084ee9c0 (DEFAULT_TABLE table=rebalance_test_table_10 [id=99374a96f2864b1f84582b7cc580e3f3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:52.717305 10623 tablet_service.cc:1467] Processing CreateTablet for tablet d8ffb869041f41259d08443b084ee9c0 (DEFAULT_TABLE table=rebalance_test_table_10 [id=99374a96f2864b1f84582b7cc580e3f3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:52.717821 10091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8ffb869041f41259d08443b084ee9c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:52.717829 10623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8ffb869041f41259d08443b084ee9c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:52.717909 10225 tablet_service.cc:1467] Processing CreateTablet for tablet 4c79cbf474df4ff0974e5611fbba66cd (DEFAULT_TABLE table=rebalance_test_table_10 [id=99374a96f2864b1f84582b7cc580e3f3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:52.717866 10491 tablet_service.cc:1467] Processing CreateTablet for tablet 4c79cbf474df4ff0974e5611fbba66cd (DEFAULT_TABLE table=rebalance_test_table_10 [id=99374a96f2864b1f84582b7cc580e3f3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:52.718336 10491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c79cbf474df4ff0974e5611fbba66cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:52.718336 10225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c79cbf474df4ff0974e5611fbba66cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:52.724164 10358 tablet_service.cc:1467] Processing CreateTablet for tablet d8ffb869041f41259d08443b084ee9c0 (DEFAULT_TABLE table=rebalance_test_table_10 [id=99374a96f2864b1f84582b7cc580e3f3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:52.724694 10358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8ffb869041f41259d08443b084ee9c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:52.729246 10758 tablet_service.cc:1467] Processing CreateTablet for tablet 4c79cbf474df4ff0974e5611fbba66cd (DEFAULT_TABLE table=rebalance_test_table_10 [id=99374a96f2864b1f84582b7cc580e3f3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:52.729760 10758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c79cbf474df4ff0974e5611fbba66cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:52.730118 10357 tablet_service.cc:1467] Processing CreateTablet for tablet e0f8668f24034001bebe8edabdf7ee9e (DEFAULT_TABLE table=rebalance_test_table_10 [id=99374a96f2864b1f84582b7cc580e3f3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:52.730736 10357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0f8668f24034001bebe8edabdf7ee9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:52.733173 11802 tablet_bootstrap.cc:492] T 4c79cbf474df4ff0974e5611fbba66cd P 34da849acb2045a9a2ef03933592dd08: Bootstrap starting.
I20240502 01:52:52.735164 11802 tablet_bootstrap.cc:654] T 4c79cbf474df4ff0974e5611fbba66cd P 34da849acb2045a9a2ef03933592dd08: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:52.738415 10624 tablet_service.cc:1467] Processing CreateTablet for tablet e0f8668f24034001bebe8edabdf7ee9e (DEFAULT_TABLE table=rebalance_test_table_10 [id=99374a96f2864b1f84582b7cc580e3f3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:52.739075 10624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0f8668f24034001bebe8edabdf7ee9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:52.739624 10092 tablet_service.cc:1467] Processing CreateTablet for tablet e0f8668f24034001bebe8edabdf7ee9e (DEFAULT_TABLE table=rebalance_test_table_10 [id=99374a96f2864b1f84582b7cc580e3f3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:52.740074 10092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0f8668f24034001bebe8edabdf7ee9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:52.753203 11803 tablet_bootstrap.cc:492] T d8ffb869041f41259d08443b084ee9c0 P a885359a82bd4515913f7b273591f851: Bootstrap starting.
I20240502 01:52:52.755954 11803 tablet_bootstrap.cc:654] T d8ffb869041f41259d08443b084ee9c0 P a885359a82bd4515913f7b273591f851: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:52.757061 11807 tablet_bootstrap.cc:492] T 4c79cbf474df4ff0974e5611fbba66cd P 7b7a47245f294ce9a264fcc8f6351c0f: Bootstrap starting.
I20240502 01:52:52.758158 11806 tablet_bootstrap.cc:492] T 4c79cbf474df4ff0974e5611fbba66cd P fd95bcd4cdcb4cc5966050ffe2cffdff: Bootstrap starting.
I20240502 01:52:52.758659 11804 tablet_bootstrap.cc:492] T d8ffb869041f41259d08443b084ee9c0 P ede1dd598e3541138a19a3ea5c052bc9: Bootstrap starting.
I20240502 01:52:52.762029 11804 tablet_bootstrap.cc:654] T d8ffb869041f41259d08443b084ee9c0 P ede1dd598e3541138a19a3ea5c052bc9: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:52.764583 11802 tablet_bootstrap.cc:492] T 4c79cbf474df4ff0974e5611fbba66cd P 34da849acb2045a9a2ef03933592dd08: No bootstrap required, opened a new log
I20240502 01:52:52.764783 11802 ts_tablet_manager.cc:1367] T 4c79cbf474df4ff0974e5611fbba66cd P 34da849acb2045a9a2ef03933592dd08: Time spent bootstrapping tablet: real 0.032s	user 0.018s	sys 0.010s
I20240502 01:52:52.765897 11802 raft_consensus.cc:348] T 4c79cbf474df4ff0974e5611fbba66cd P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } } peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } }
I20240502 01:52:52.766153 11802 raft_consensus.cc:374] T 4c79cbf474df4ff0974e5611fbba66cd P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:52.766265 11802 raft_consensus.cc:729] T 4c79cbf474df4ff0974e5611fbba66cd P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34da849acb2045a9a2ef03933592dd08, State: Initialized, Role: FOLLOWER
I20240502 01:52:52.766542 11802 consensus_queue.cc:260] T 4c79cbf474df4ff0974e5611fbba66cd P 34da849acb2045a9a2ef03933592dd08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } } peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } }
I20240502 01:52:52.767371 11802 ts_tablet_manager.cc:1398] T 4c79cbf474df4ff0974e5611fbba66cd P 34da849acb2045a9a2ef03933592dd08: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.003s
I20240502 01:52:52.768575 11807 tablet_bootstrap.cc:654] T 4c79cbf474df4ff0974e5611fbba66cd P 7b7a47245f294ce9a264fcc8f6351c0f: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:52.770006 11806 tablet_bootstrap.cc:654] T 4c79cbf474df4ff0974e5611fbba66cd P fd95bcd4cdcb4cc5966050ffe2cffdff: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:52.774847 11801 tablet_bootstrap.cc:492] T d8ffb869041f41259d08443b084ee9c0 P a8c4a518929d4bb3a88bdfb1649493bd: Bootstrap starting.
I20240502 01:52:52.775751 11807 tablet_bootstrap.cc:492] T 4c79cbf474df4ff0974e5611fbba66cd P 7b7a47245f294ce9a264fcc8f6351c0f: No bootstrap required, opened a new log
I20240502 01:52:52.775925 11807 ts_tablet_manager.cc:1367] T 4c79cbf474df4ff0974e5611fbba66cd P 7b7a47245f294ce9a264fcc8f6351c0f: Time spent bootstrapping tablet: real 0.019s	user 0.003s	sys 0.004s
I20240502 01:52:52.776903 11801 tablet_bootstrap.cc:654] T d8ffb869041f41259d08443b084ee9c0 P a8c4a518929d4bb3a88bdfb1649493bd: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:52.778249 11803 tablet_bootstrap.cc:492] T d8ffb869041f41259d08443b084ee9c0 P a885359a82bd4515913f7b273591f851: No bootstrap required, opened a new log
I20240502 01:52:52.778679 11803 ts_tablet_manager.cc:1367] T d8ffb869041f41259d08443b084ee9c0 P a885359a82bd4515913f7b273591f851: Time spent bootstrapping tablet: real 0.026s	user 0.000s	sys 0.005s
I20240502 01:52:52.779266 11807 raft_consensus.cc:348] T 4c79cbf474df4ff0974e5611fbba66cd P 7b7a47245f294ce9a264fcc8f6351c0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } } peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } }
I20240502 01:52:52.779621 11807 raft_consensus.cc:374] T 4c79cbf474df4ff0974e5611fbba66cd P 7b7a47245f294ce9a264fcc8f6351c0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:52.779758 11807 raft_consensus.cc:729] T 4c79cbf474df4ff0974e5611fbba66cd P 7b7a47245f294ce9a264fcc8f6351c0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b7a47245f294ce9a264fcc8f6351c0f, State: Initialized, Role: FOLLOWER
I20240502 01:52:52.780011 11803 raft_consensus.cc:348] T d8ffb869041f41259d08443b084ee9c0 P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } }
I20240502 01:52:52.780344 11803 raft_consensus.cc:374] T d8ffb869041f41259d08443b084ee9c0 P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:52.780512 11803 raft_consensus.cc:729] T d8ffb869041f41259d08443b084ee9c0 P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a885359a82bd4515913f7b273591f851, State: Initialized, Role: FOLLOWER
I20240502 01:52:52.780300 11807 consensus_queue.cc:260] T 4c79cbf474df4ff0974e5611fbba66cd P 7b7a47245f294ce9a264fcc8f6351c0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } } peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } }
I20240502 01:52:52.781021 11803 consensus_queue.cc:260] T d8ffb869041f41259d08443b084ee9c0 P a885359a82bd4515913f7b273591f851 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } }
I20240502 01:52:52.781845 11803 ts_tablet_manager.cc:1398] T d8ffb869041f41259d08443b084ee9c0 P a885359a82bd4515913f7b273591f851: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.002s
I20240502 01:52:52.782896 11804 tablet_bootstrap.cc:492] T d8ffb869041f41259d08443b084ee9c0 P ede1dd598e3541138a19a3ea5c052bc9: No bootstrap required, opened a new log
I20240502 01:52:52.783066 11803 tablet_bootstrap.cc:492] T e0f8668f24034001bebe8edabdf7ee9e P a885359a82bd4515913f7b273591f851: Bootstrap starting.
I20240502 01:52:52.783094 11804 ts_tablet_manager.cc:1367] T d8ffb869041f41259d08443b084ee9c0 P ede1dd598e3541138a19a3ea5c052bc9: Time spent bootstrapping tablet: real 0.025s	user 0.002s	sys 0.002s
I20240502 01:52:52.784157 11804 raft_consensus.cc:348] T d8ffb869041f41259d08443b084ee9c0 P ede1dd598e3541138a19a3ea5c052bc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } }
I20240502 01:52:52.784440 11804 raft_consensus.cc:374] T d8ffb869041f41259d08443b084ee9c0 P ede1dd598e3541138a19a3ea5c052bc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:52.784575 11804 raft_consensus.cc:729] T d8ffb869041f41259d08443b084ee9c0 P ede1dd598e3541138a19a3ea5c052bc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ede1dd598e3541138a19a3ea5c052bc9, State: Initialized, Role: FOLLOWER
I20240502 01:52:52.784854 11804 consensus_queue.cc:260] T d8ffb869041f41259d08443b084ee9c0 P ede1dd598e3541138a19a3ea5c052bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } }
I20240502 01:52:52.785547 11803 tablet_bootstrap.cc:654] T e0f8668f24034001bebe8edabdf7ee9e P a885359a82bd4515913f7b273591f851: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:52.786530 11804 ts_tablet_manager.cc:1398] T d8ffb869041f41259d08443b084ee9c0 P ede1dd598e3541138a19a3ea5c052bc9: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 01:52:52.786882 11804 tablet_bootstrap.cc:492] T e0f8668f24034001bebe8edabdf7ee9e P ede1dd598e3541138a19a3ea5c052bc9: Bootstrap starting.
I20240502 01:52:52.788753 11804 tablet_bootstrap.cc:654] T e0f8668f24034001bebe8edabdf7ee9e P ede1dd598e3541138a19a3ea5c052bc9: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:52.792295 11806 tablet_bootstrap.cc:492] T 4c79cbf474df4ff0974e5611fbba66cd P fd95bcd4cdcb4cc5966050ffe2cffdff: No bootstrap required, opened a new log
I20240502 01:52:52.792610 11806 ts_tablet_manager.cc:1367] T 4c79cbf474df4ff0974e5611fbba66cd P fd95bcd4cdcb4cc5966050ffe2cffdff: Time spent bootstrapping tablet: real 0.035s	user 0.003s	sys 0.004s
I20240502 01:52:52.792799 11801 tablet_bootstrap.cc:492] T d8ffb869041f41259d08443b084ee9c0 P a8c4a518929d4bb3a88bdfb1649493bd: No bootstrap required, opened a new log
I20240502 01:52:52.793030 11801 ts_tablet_manager.cc:1367] T d8ffb869041f41259d08443b084ee9c0 P a8c4a518929d4bb3a88bdfb1649493bd: Time spent bootstrapping tablet: real 0.019s	user 0.004s	sys 0.008s
I20240502 01:52:52.793823 11807 ts_tablet_manager.cc:1398] T 4c79cbf474df4ff0974e5611fbba66cd P 7b7a47245f294ce9a264fcc8f6351c0f: Time spent starting tablet: real 0.018s	user 0.003s	sys 0.000s
I20240502 01:52:52.793766 11806 raft_consensus.cc:348] T 4c79cbf474df4ff0974e5611fbba66cd P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } } peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } }
I20240502 01:52:52.794168 11806 raft_consensus.cc:374] T 4c79cbf474df4ff0974e5611fbba66cd P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:52.794314 11806 raft_consensus.cc:729] T 4c79cbf474df4ff0974e5611fbba66cd P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd95bcd4cdcb4cc5966050ffe2cffdff, State: Initialized, Role: FOLLOWER
I20240502 01:52:52.794669 11806 consensus_queue.cc:260] T 4c79cbf474df4ff0974e5611fbba66cd P fd95bcd4cdcb4cc5966050ffe2cffdff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } } peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } }
I20240502 01:52:52.794917 11801 raft_consensus.cc:348] T d8ffb869041f41259d08443b084ee9c0 P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } }
I20240502 01:52:52.795217 11801 raft_consensus.cc:374] T d8ffb869041f41259d08443b084ee9c0 P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:52.795372 11801 raft_consensus.cc:729] T d8ffb869041f41259d08443b084ee9c0 P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8c4a518929d4bb3a88bdfb1649493bd, State: Initialized, Role: FOLLOWER
I20240502 01:52:52.796001 11801 consensus_queue.cc:260] T d8ffb869041f41259d08443b084ee9c0 P a8c4a518929d4bb3a88bdfb1649493bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } }
I20240502 01:52:52.797235 11801 ts_tablet_manager.cc:1398] T d8ffb869041f41259d08443b084ee9c0 P a8c4a518929d4bb3a88bdfb1649493bd: Time spent starting tablet: real 0.004s	user 0.001s	sys 0.002s
I20240502 01:52:52.797633 11801 tablet_bootstrap.cc:492] T e0f8668f24034001bebe8edabdf7ee9e P a8c4a518929d4bb3a88bdfb1649493bd: Bootstrap starting.
I20240502 01:52:52.802354 11801 tablet_bootstrap.cc:654] T e0f8668f24034001bebe8edabdf7ee9e P a8c4a518929d4bb3a88bdfb1649493bd: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:52.805136 11806 ts_tablet_manager.cc:1398] T 4c79cbf474df4ff0974e5611fbba66cd P fd95bcd4cdcb4cc5966050ffe2cffdff: Time spent starting tablet: real 0.012s	user 0.000s	sys 0.002s
I20240502 01:52:52.807360 11804 tablet_bootstrap.cc:492] T e0f8668f24034001bebe8edabdf7ee9e P ede1dd598e3541138a19a3ea5c052bc9: No bootstrap required, opened a new log
I20240502 01:52:52.807559 11750 raft_consensus.cc:482] T d8ffb869041f41259d08443b084ee9c0 P ede1dd598e3541138a19a3ea5c052bc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:52.807631 11804 ts_tablet_manager.cc:1367] T e0f8668f24034001bebe8edabdf7ee9e P ede1dd598e3541138a19a3ea5c052bc9: Time spent bootstrapping tablet: real 0.021s	user 0.005s	sys 0.007s
I20240502 01:52:52.807751 11750 raft_consensus.cc:504] T d8ffb869041f41259d08443b084ee9c0 P ede1dd598e3541138a19a3ea5c052bc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } }
I20240502 01:52:52.808598 11750 leader_election.cc:288] T d8ffb869041f41259d08443b084ee9c0 P ede1dd598e3541138a19a3ea5c052bc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a8c4a518929d4bb3a88bdfb1649493bd (127.9.176.197:44061), a885359a82bd4515913f7b273591f851 (127.9.176.193:38373)
I20240502 01:52:52.808651 11804 raft_consensus.cc:348] T e0f8668f24034001bebe8edabdf7ee9e P ede1dd598e3541138a19a3ea5c052bc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } }
I20240502 01:52:52.808898 11804 raft_consensus.cc:374] T e0f8668f24034001bebe8edabdf7ee9e P ede1dd598e3541138a19a3ea5c052bc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:52.808986 11804 raft_consensus.cc:729] T e0f8668f24034001bebe8edabdf7ee9e P ede1dd598e3541138a19a3ea5c052bc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ede1dd598e3541138a19a3ea5c052bc9, State: Initialized, Role: FOLLOWER
I20240502 01:52:52.809119 10644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8ffb869041f41259d08443b084ee9c0" candidate_uuid: "ede1dd598e3541138a19a3ea5c052bc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" is_pre_election: true
I20240502 01:52:52.809269 11804 consensus_queue.cc:260] T e0f8668f24034001bebe8edabdf7ee9e P ede1dd598e3541138a19a3ea5c052bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } }
I20240502 01:52:52.809432 10644 raft_consensus.cc:2454] T d8ffb869041f41259d08443b084ee9c0 P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ede1dd598e3541138a19a3ea5c052bc9 in term 0.
I20240502 01:52:52.809367 10112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8ffb869041f41259d08443b084ee9c0" candidate_uuid: "ede1dd598e3541138a19a3ea5c052bc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a885359a82bd4515913f7b273591f851" is_pre_election: true
I20240502 01:52:52.809600 10112 raft_consensus.cc:2454] T d8ffb869041f41259d08443b084ee9c0 P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ede1dd598e3541138a19a3ea5c052bc9 in term 0.
I20240502 01:52:52.810065 10313 leader_election.cc:302] T d8ffb869041f41259d08443b084ee9c0 P ede1dd598e3541138a19a3ea5c052bc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a8c4a518929d4bb3a88bdfb1649493bd, ede1dd598e3541138a19a3ea5c052bc9; no voters: 
I20240502 01:52:52.810700 11750 raft_consensus.cc:2781] T d8ffb869041f41259d08443b084ee9c0 P ede1dd598e3541138a19a3ea5c052bc9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:52.810833 11750 raft_consensus.cc:482] T d8ffb869041f41259d08443b084ee9c0 P ede1dd598e3541138a19a3ea5c052bc9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:52.810935 11750 raft_consensus.cc:3037] T d8ffb869041f41259d08443b084ee9c0 P ede1dd598e3541138a19a3ea5c052bc9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:52.811522 11803 tablet_bootstrap.cc:492] T e0f8668f24034001bebe8edabdf7ee9e P a885359a82bd4515913f7b273591f851: No bootstrap required, opened a new log
I20240502 01:52:52.811746 11803 ts_tablet_manager.cc:1367] T e0f8668f24034001bebe8edabdf7ee9e P a885359a82bd4515913f7b273591f851: Time spent bootstrapping tablet: real 0.029s	user 0.010s	sys 0.000s
I20240502 01:52:52.812724 11803 raft_consensus.cc:348] T e0f8668f24034001bebe8edabdf7ee9e P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } }
I20240502 01:52:52.813150 11803 raft_consensus.cc:374] T e0f8668f24034001bebe8edabdf7ee9e P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:52.813308 11803 raft_consensus.cc:729] T e0f8668f24034001bebe8edabdf7ee9e P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a885359a82bd4515913f7b273591f851, State: Initialized, Role: FOLLOWER
I20240502 01:52:52.815248 11803 consensus_queue.cc:260] T e0f8668f24034001bebe8edabdf7ee9e P a885359a82bd4515913f7b273591f851 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } }
I20240502 01:52:52.816005 11750 raft_consensus.cc:504] T d8ffb869041f41259d08443b084ee9c0 P ede1dd598e3541138a19a3ea5c052bc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } }
I20240502 01:52:52.816395 11803 ts_tablet_manager.cc:1398] T e0f8668f24034001bebe8edabdf7ee9e P a885359a82bd4515913f7b273591f851: Time spent starting tablet: real 0.005s	user 0.003s	sys 0.000s
I20240502 01:52:52.817524 10644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8ffb869041f41259d08443b084ee9c0" candidate_uuid: "ede1dd598e3541138a19a3ea5c052bc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8c4a518929d4bb3a88bdfb1649493bd"
I20240502 01:52:52.817658 11801 tablet_bootstrap.cc:492] T e0f8668f24034001bebe8edabdf7ee9e P a8c4a518929d4bb3a88bdfb1649493bd: No bootstrap required, opened a new log
I20240502 01:52:52.817791 10644 raft_consensus.cc:3037] T d8ffb869041f41259d08443b084ee9c0 P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:52.817831 11801 ts_tablet_manager.cc:1367] T e0f8668f24034001bebe8edabdf7ee9e P a8c4a518929d4bb3a88bdfb1649493bd: Time spent bootstrapping tablet: real 0.020s	user 0.002s	sys 0.003s
I20240502 01:52:52.817821 11750 leader_election.cc:288] T d8ffb869041f41259d08443b084ee9c0 P ede1dd598e3541138a19a3ea5c052bc9 [CANDIDATE]: Term 1 election: Requested vote from peers a8c4a518929d4bb3a88bdfb1649493bd (127.9.176.197:44061), a885359a82bd4515913f7b273591f851 (127.9.176.193:38373)
I20240502 01:52:52.818848 11801 raft_consensus.cc:348] T e0f8668f24034001bebe8edabdf7ee9e P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } }
I20240502 01:52:52.819095 11801 raft_consensus.cc:374] T e0f8668f24034001bebe8edabdf7ee9e P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:52.819167 11801 raft_consensus.cc:729] T e0f8668f24034001bebe8edabdf7ee9e P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8c4a518929d4bb3a88bdfb1649493bd, State: Initialized, Role: FOLLOWER
I20240502 01:52:52.819427 11801 consensus_queue.cc:260] T e0f8668f24034001bebe8edabdf7ee9e P a8c4a518929d4bb3a88bdfb1649493bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } }
I20240502 01:52:52.810074 11804 ts_tablet_manager.cc:1398] T e0f8668f24034001bebe8edabdf7ee9e P ede1dd598e3541138a19a3ea5c052bc9: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240502 01:52:52.820106 11801 ts_tablet_manager.cc:1398] T e0f8668f24034001bebe8edabdf7ee9e P a8c4a518929d4bb3a88bdfb1649493bd: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.000s
I20240502 01:52:52.822171 10112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8ffb869041f41259d08443b084ee9c0" candidate_uuid: "ede1dd598e3541138a19a3ea5c052bc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a885359a82bd4515913f7b273591f851"
I20240502 01:52:52.822481 10112 raft_consensus.cc:3037] T d8ffb869041f41259d08443b084ee9c0 P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:52.823279 10644 raft_consensus.cc:2454] T d8ffb869041f41259d08443b084ee9c0 P a8c4a518929d4bb3a88bdfb1649493bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ede1dd598e3541138a19a3ea5c052bc9 in term 1.
I20240502 01:52:52.824178 10313 leader_election.cc:302] T d8ffb869041f41259d08443b084ee9c0 P ede1dd598e3541138a19a3ea5c052bc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a8c4a518929d4bb3a88bdfb1649493bd, ede1dd598e3541138a19a3ea5c052bc9; no voters: 
I20240502 01:52:52.826310 10112 raft_consensus.cc:2454] T d8ffb869041f41259d08443b084ee9c0 P a885359a82bd4515913f7b273591f851 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ede1dd598e3541138a19a3ea5c052bc9 in term 1.
I20240502 01:52:52.826359 11750 raft_consensus.cc:2781] T d8ffb869041f41259d08443b084ee9c0 P ede1dd598e3541138a19a3ea5c052bc9 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:52.828032 11750 raft_consensus.cc:686] T d8ffb869041f41259d08443b084ee9c0 P ede1dd598e3541138a19a3ea5c052bc9 [term 1 LEADER]: Becoming Leader. State: Replica: ede1dd598e3541138a19a3ea5c052bc9, State: Running, Role: LEADER
I20240502 01:52:52.828419 11750 consensus_queue.cc:237] T d8ffb869041f41259d08443b084ee9c0 P ede1dd598e3541138a19a3ea5c052bc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } }
I20240502 01:52:52.831671  9961 catalog_manager.cc:5484] T d8ffb869041f41259d08443b084ee9c0 P ede1dd598e3541138a19a3ea5c052bc9 reported cstate change: term changed from 0 to 1, leader changed from <none> to ede1dd598e3541138a19a3ea5c052bc9 (127.9.176.195). New cstate: current_term: 1 leader_uuid: "ede1dd598e3541138a19a3ea5c052bc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } health_report { overall_health: UNKNOWN } } }
I20240502 01:52:52.931836 11724 raft_consensus.cc:482] T 4c79cbf474df4ff0974e5611fbba66cd P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:52.932056 11724 raft_consensus.cc:504] T 4c79cbf474df4ff0974e5611fbba66cd P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } } peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } }
I20240502 01:52:52.932793 11724 leader_election.cc:288] T 4c79cbf474df4ff0974e5611fbba66cd P 34da849acb2045a9a2ef03933592dd08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b7a47245f294ce9a264fcc8f6351c0f (127.9.176.194:43645), fd95bcd4cdcb4cc5966050ffe2cffdff (127.9.176.198:33039)
I20240502 01:52:52.933367 10244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c79cbf474df4ff0974e5611fbba66cd" candidate_uuid: "34da849acb2045a9a2ef03933592dd08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" is_pre_election: true
I20240502 01:52:52.933405 10779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c79cbf474df4ff0974e5611fbba66cd" candidate_uuid: "34da849acb2045a9a2ef03933592dd08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" is_pre_election: true
I20240502 01:52:52.933590 10244 raft_consensus.cc:2454] T 4c79cbf474df4ff0974e5611fbba66cd P 7b7a47245f294ce9a264fcc8f6351c0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34da849acb2045a9a2ef03933592dd08 in term 0.
I20240502 01:52:52.933648 10779 raft_consensus.cc:2454] T 4c79cbf474df4ff0974e5611fbba66cd P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34da849acb2045a9a2ef03933592dd08 in term 0.
I20240502 01:52:52.934119 10443 leader_election.cc:302] T 4c79cbf474df4ff0974e5611fbba66cd P 34da849acb2045a9a2ef03933592dd08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34da849acb2045a9a2ef03933592dd08, 7b7a47245f294ce9a264fcc8f6351c0f; no voters: 
I20240502 01:52:52.934425 11724 raft_consensus.cc:2781] T 4c79cbf474df4ff0974e5611fbba66cd P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:52.934549 11724 raft_consensus.cc:482] T 4c79cbf474df4ff0974e5611fbba66cd P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:52.934643 11724 raft_consensus.cc:3037] T 4c79cbf474df4ff0974e5611fbba66cd P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:52.936919 11724 raft_consensus.cc:504] T 4c79cbf474df4ff0974e5611fbba66cd P 34da849acb2045a9a2ef03933592dd08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } } peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } }
I20240502 01:52:52.937557 11724 leader_election.cc:288] T 4c79cbf474df4ff0974e5611fbba66cd P 34da849acb2045a9a2ef03933592dd08 [CANDIDATE]: Term 1 election: Requested vote from peers 7b7a47245f294ce9a264fcc8f6351c0f (127.9.176.194:43645), fd95bcd4cdcb4cc5966050ffe2cffdff (127.9.176.198:33039)
I20240502 01:52:52.938123 10779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c79cbf474df4ff0974e5611fbba66cd" candidate_uuid: "34da849acb2045a9a2ef03933592dd08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff"
I20240502 01:52:52.938143 10244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c79cbf474df4ff0974e5611fbba66cd" candidate_uuid: "34da849acb2045a9a2ef03933592dd08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b7a47245f294ce9a264fcc8f6351c0f"
I20240502 01:52:52.938338 10244 raft_consensus.cc:3037] T 4c79cbf474df4ff0974e5611fbba66cd P 7b7a47245f294ce9a264fcc8f6351c0f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:52.938333 10779 raft_consensus.cc:3037] T 4c79cbf474df4ff0974e5611fbba66cd P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:52.940333 10779 raft_consensus.cc:2454] T 4c79cbf474df4ff0974e5611fbba66cd P fd95bcd4cdcb4cc5966050ffe2cffdff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34da849acb2045a9a2ef03933592dd08 in term 1.
I20240502 01:52:52.940362 10244 raft_consensus.cc:2454] T 4c79cbf474df4ff0974e5611fbba66cd P 7b7a47245f294ce9a264fcc8f6351c0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34da849acb2045a9a2ef03933592dd08 in term 1.
I20240502 01:52:52.940871 10443 leader_election.cc:302] T 4c79cbf474df4ff0974e5611fbba66cd P 34da849acb2045a9a2ef03933592dd08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34da849acb2045a9a2ef03933592dd08, 7b7a47245f294ce9a264fcc8f6351c0f; no voters: 
I20240502 01:52:52.941224 11724 raft_consensus.cc:2781] T 4c79cbf474df4ff0974e5611fbba66cd P 34da849acb2045a9a2ef03933592dd08 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:52.941481 11724 raft_consensus.cc:686] T 4c79cbf474df4ff0974e5611fbba66cd P 34da849acb2045a9a2ef03933592dd08 [term 1 LEADER]: Becoming Leader. State: Replica: 34da849acb2045a9a2ef03933592dd08, State: Running, Role: LEADER
I20240502 01:52:52.941854 11724 consensus_queue.cc:237] T 4c79cbf474df4ff0974e5611fbba66cd P 34da849acb2045a9a2ef03933592dd08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } } peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } }
I20240502 01:52:52.945042  9961 catalog_manager.cc:5484] T 4c79cbf474df4ff0974e5611fbba66cd P 34da849acb2045a9a2ef03933592dd08 reported cstate change: term changed from 0 to 1, leader changed from <none> to 34da849acb2045a9a2ef03933592dd08 (127.9.176.196). New cstate: current_term: 1 leader_uuid: "34da849acb2045a9a2ef03933592dd08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } health_report { overall_health: UNKNOWN } } }
I20240502 01:52:52.973968 11647 raft_consensus.cc:482] T e0f8668f24034001bebe8edabdf7ee9e P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:52.974220 11647 raft_consensus.cc:504] T e0f8668f24034001bebe8edabdf7ee9e P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } }
I20240502 01:52:52.975095 11647 leader_election.cc:288] T e0f8668f24034001bebe8edabdf7ee9e P a885359a82bd4515913f7b273591f851 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ede1dd598e3541138a19a3ea5c052bc9 (127.9.176.195:38701), a8c4a518929d4bb3a88bdfb1649493bd (127.9.176.197:44061)
I20240502 01:52:52.975713 10644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f8668f24034001bebe8edabdf7ee9e" candidate_uuid: "a885359a82bd4515913f7b273591f851" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" is_pre_election: true
I20240502 01:52:52.975713 10376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f8668f24034001bebe8edabdf7ee9e" candidate_uuid: "a885359a82bd4515913f7b273591f851" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ede1dd598e3541138a19a3ea5c052bc9" is_pre_election: true
I20240502 01:52:52.975970 10644 raft_consensus.cc:2454] T e0f8668f24034001bebe8edabdf7ee9e P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a885359a82bd4515913f7b273591f851 in term 0.
I20240502 01:52:52.975970 10376 raft_consensus.cc:2454] T e0f8668f24034001bebe8edabdf7ee9e P ede1dd598e3541138a19a3ea5c052bc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a885359a82bd4515913f7b273591f851 in term 0.
I20240502 01:52:52.976465 10045 leader_election.cc:302] T e0f8668f24034001bebe8edabdf7ee9e P a885359a82bd4515913f7b273591f851 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: a885359a82bd4515913f7b273591f851, a8c4a518929d4bb3a88bdfb1649493bd, ede1dd598e3541138a19a3ea5c052bc9; no voters: 
I20240502 01:52:52.976783 11647 raft_consensus.cc:2781] T e0f8668f24034001bebe8edabdf7ee9e P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:52.976900 11647 raft_consensus.cc:482] T e0f8668f24034001bebe8edabdf7ee9e P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:52.976998 11647 raft_consensus.cc:3037] T e0f8668f24034001bebe8edabdf7ee9e P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:52.979140 11647 raft_consensus.cc:504] T e0f8668f24034001bebe8edabdf7ee9e P a885359a82bd4515913f7b273591f851 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } }
I20240502 01:52:52.979878 11647 leader_election.cc:288] T e0f8668f24034001bebe8edabdf7ee9e P a885359a82bd4515913f7b273591f851 [CANDIDATE]: Term 1 election: Requested vote from peers ede1dd598e3541138a19a3ea5c052bc9 (127.9.176.195:38701), a8c4a518929d4bb3a88bdfb1649493bd (127.9.176.197:44061)
I20240502 01:52:52.980492 10644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f8668f24034001bebe8edabdf7ee9e" candidate_uuid: "a885359a82bd4515913f7b273591f851" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8c4a518929d4bb3a88bdfb1649493bd"
I20240502 01:52:52.980597 10376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f8668f24034001bebe8edabdf7ee9e" candidate_uuid: "a885359a82bd4515913f7b273591f851" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ede1dd598e3541138a19a3ea5c052bc9"
I20240502 01:52:52.980719 10644 raft_consensus.cc:3037] T e0f8668f24034001bebe8edabdf7ee9e P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:52.980829 10376 raft_consensus.cc:3037] T e0f8668f24034001bebe8edabdf7ee9e P ede1dd598e3541138a19a3ea5c052bc9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:52.982913 10376 raft_consensus.cc:2454] T e0f8668f24034001bebe8edabdf7ee9e P ede1dd598e3541138a19a3ea5c052bc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a885359a82bd4515913f7b273591f851 in term 1.
I20240502 01:52:52.982919 10644 raft_consensus.cc:2454] T e0f8668f24034001bebe8edabdf7ee9e P a8c4a518929d4bb3a88bdfb1649493bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a885359a82bd4515913f7b273591f851 in term 1.
I20240502 01:52:52.983438 10045 leader_election.cc:302] T e0f8668f24034001bebe8edabdf7ee9e P a885359a82bd4515913f7b273591f851 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a885359a82bd4515913f7b273591f851, ede1dd598e3541138a19a3ea5c052bc9; no voters: 
I20240502 01:52:52.983942 11647 raft_consensus.cc:2781] T e0f8668f24034001bebe8edabdf7ee9e P a885359a82bd4515913f7b273591f851 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:52.984108 11647 raft_consensus.cc:686] T e0f8668f24034001bebe8edabdf7ee9e P a885359a82bd4515913f7b273591f851 [term 1 LEADER]: Becoming Leader. State: Replica: a885359a82bd4515913f7b273591f851, State: Running, Role: LEADER
I20240502 01:52:52.984422 11647 consensus_queue.cc:237] T e0f8668f24034001bebe8edabdf7ee9e P a885359a82bd4515913f7b273591f851 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } }
I20240502 01:52:52.988374  9961 catalog_manager.cc:5484] T e0f8668f24034001bebe8edabdf7ee9e P a885359a82bd4515913f7b273591f851 reported cstate change: term changed from 0 to 1, leader changed from <none> to a885359a82bd4515913f7b273591f851 (127.9.176.193). New cstate: current_term: 1 leader_uuid: "a885359a82bd4515913f7b273591f851" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } health_report { overall_health: HEALTHY } } }
I20240502 01:52:53.247848 11750 consensus_queue.cc:1035] T d8ffb869041f41259d08443b084ee9c0 P ede1dd598e3541138a19a3ea5c052bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:53.256004 11816 consensus_queue.cc:1035] T d8ffb869041f41259d08443b084ee9c0 P ede1dd598e3541138a19a3ea5c052bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:52:53.424696 11820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:53.425052 11820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:53.432984 11820 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_10'
I20240502 01:52:53.503762 11724 consensus_queue.cc:1035] T 4c79cbf474df4ff0974e5611fbba66cd P 34da849acb2045a9a2ef03933592dd08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:53.583688 11647 consensus_queue.cc:1035] T e0f8668f24034001bebe8edabdf7ee9e P a885359a82bd4515913f7b273591f851 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:53.593164 11647 consensus_queue.cc:1035] T e0f8668f24034001bebe8edabdf7ee9e P a885359a82bd4515913f7b273591f851 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:53.637821 11856 consensus_queue.cc:1035] T 4c79cbf474df4ff0974e5611fbba66cd P 34da849acb2045a9a2ef03933592dd08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s

INSERT report
    rows total: 2000
    time total: 470.158 ms
  time per row: 0.235079 ms
I20240502 01:52:54.067895  9963 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56524:
name: "rebalance_test_table_11"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 01:52:54.078855 10758 tablet_service.cc:1467] Processing CreateTablet for tablet f7042550f3fe4a4780566f2fbdda3678 (DEFAULT_TABLE table=rebalance_test_table_11 [id=c21f10c7b6da46ea83164f2705685091]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:54.079023 10491 tablet_service.cc:1467] Processing CreateTablet for tablet f7042550f3fe4a4780566f2fbdda3678 (DEFAULT_TABLE table=rebalance_test_table_11 [id=c21f10c7b6da46ea83164f2705685091]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:54.079428 10758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7042550f3fe4a4780566f2fbdda3678. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:54.079519 10491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7042550f3fe4a4780566f2fbdda3678. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:54.079994 10492 tablet_service.cc:1467] Processing CreateTablet for tablet e71b5c9725d9400d8ed9faad7276a474 (DEFAULT_TABLE table=rebalance_test_table_11 [id=c21f10c7b6da46ea83164f2705685091]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:54.080338 10759 tablet_service.cc:1467] Processing CreateTablet for tablet e71b5c9725d9400d8ed9faad7276a474 (DEFAULT_TABLE table=rebalance_test_table_11 [id=c21f10c7b6da46ea83164f2705685091]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:54.080435 10492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e71b5c9725d9400d8ed9faad7276a474. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:54.080761 10759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e71b5c9725d9400d8ed9faad7276a474. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:54.085003 10225 tablet_service.cc:1467] Processing CreateTablet for tablet f7042550f3fe4a4780566f2fbdda3678 (DEFAULT_TABLE table=rebalance_test_table_11 [id=c21f10c7b6da46ea83164f2705685091]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:54.085522 10225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7042550f3fe4a4780566f2fbdda3678. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:54.087684 10224 tablet_service.cc:1467] Processing CreateTablet for tablet e71b5c9725d9400d8ed9faad7276a474 (DEFAULT_TABLE table=rebalance_test_table_11 [id=c21f10c7b6da46ea83164f2705685091]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:54.087684 10091 tablet_service.cc:1467] Processing CreateTablet for tablet d4384df5dd744d399034ef24e55fbbf9 (DEFAULT_TABLE table=rebalance_test_table_11 [id=c21f10c7b6da46ea83164f2705685091]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:54.088177 10224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e71b5c9725d9400d8ed9faad7276a474. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:54.088166 10091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4384df5dd744d399034ef24e55fbbf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:54.091765 10358 tablet_service.cc:1467] Processing CreateTablet for tablet d4384df5dd744d399034ef24e55fbbf9 (DEFAULT_TABLE table=rebalance_test_table_11 [id=c21f10c7b6da46ea83164f2705685091]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:54.092304 10358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4384df5dd744d399034ef24e55fbbf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:54.096643 10623 tablet_service.cc:1467] Processing CreateTablet for tablet d4384df5dd744d399034ef24e55fbbf9 (DEFAULT_TABLE table=rebalance_test_table_11 [id=c21f10c7b6da46ea83164f2705685091]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:54.097174 10623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4384df5dd744d399034ef24e55fbbf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:54.121841 11897 tablet_bootstrap.cc:492] T d4384df5dd744d399034ef24e55fbbf9 P a8c4a518929d4bb3a88bdfb1649493bd: Bootstrap starting.
I20240502 01:52:54.122598 11898 tablet_bootstrap.cc:492] T e71b5c9725d9400d8ed9faad7276a474 P 7b7a47245f294ce9a264fcc8f6351c0f: Bootstrap starting.
I20240502 01:52:54.125880 11899 tablet_bootstrap.cc:492] T e71b5c9725d9400d8ed9faad7276a474 P fd95bcd4cdcb4cc5966050ffe2cffdff: Bootstrap starting.
I20240502 01:52:54.126478 11897 tablet_bootstrap.cc:654] T d4384df5dd744d399034ef24e55fbbf9 P a8c4a518929d4bb3a88bdfb1649493bd: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:54.126602 11900 tablet_bootstrap.cc:492] T d4384df5dd744d399034ef24e55fbbf9 P ede1dd598e3541138a19a3ea5c052bc9: Bootstrap starting.
I20240502 01:52:54.128185 11898 tablet_bootstrap.cc:654] T e71b5c9725d9400d8ed9faad7276a474 P 7b7a47245f294ce9a264fcc8f6351c0f: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:54.130352 11899 tablet_bootstrap.cc:654] T e71b5c9725d9400d8ed9faad7276a474 P fd95bcd4cdcb4cc5966050ffe2cffdff: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:54.132367 11896 tablet_bootstrap.cc:492] T d4384df5dd744d399034ef24e55fbbf9 P a885359a82bd4515913f7b273591f851: Bootstrap starting.
I20240502 01:52:54.133615 11895 tablet_bootstrap.cc:492] T e71b5c9725d9400d8ed9faad7276a474 P 34da849acb2045a9a2ef03933592dd08: Bootstrap starting.
I20240502 01:52:54.133764 11900 tablet_bootstrap.cc:654] T d4384df5dd744d399034ef24e55fbbf9 P ede1dd598e3541138a19a3ea5c052bc9: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:54.134461 11898 tablet_bootstrap.cc:492] T e71b5c9725d9400d8ed9faad7276a474 P 7b7a47245f294ce9a264fcc8f6351c0f: No bootstrap required, opened a new log
I20240502 01:52:54.134706 11898 ts_tablet_manager.cc:1367] T e71b5c9725d9400d8ed9faad7276a474 P 7b7a47245f294ce9a264fcc8f6351c0f: Time spent bootstrapping tablet: real 0.012s	user 0.005s	sys 0.000s
I20240502 01:52:54.135831 11896 tablet_bootstrap.cc:654] T d4384df5dd744d399034ef24e55fbbf9 P a885359a82bd4515913f7b273591f851: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:54.136144 11898 raft_consensus.cc:348] T e71b5c9725d9400d8ed9faad7276a474 P 7b7a47245f294ce9a264fcc8f6351c0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } }
I20240502 01:52:54.136950 11898 raft_consensus.cc:374] T e71b5c9725d9400d8ed9faad7276a474 P 7b7a47245f294ce9a264fcc8f6351c0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:54.139680 11898 raft_consensus.cc:729] T e71b5c9725d9400d8ed9faad7276a474 P 7b7a47245f294ce9a264fcc8f6351c0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b7a47245f294ce9a264fcc8f6351c0f, State: Initialized, Role: FOLLOWER
I20240502 01:52:54.140146 11898 consensus_queue.cc:260] T e71b5c9725d9400d8ed9faad7276a474 P 7b7a47245f294ce9a264fcc8f6351c0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } }
I20240502 01:52:54.143013 11898 ts_tablet_manager.cc:1398] T e71b5c9725d9400d8ed9faad7276a474 P 7b7a47245f294ce9a264fcc8f6351c0f: Time spent starting tablet: real 0.008s	user 0.002s	sys 0.000s
I20240502 01:52:54.143596 11896 tablet_bootstrap.cc:492] T d4384df5dd744d399034ef24e55fbbf9 P a885359a82bd4515913f7b273591f851: No bootstrap required, opened a new log
I20240502 01:52:54.143781 11896 ts_tablet_manager.cc:1367] T d4384df5dd744d399034ef24e55fbbf9 P a885359a82bd4515913f7b273591f851: Time spent bootstrapping tablet: real 0.012s	user 0.003s	sys 0.003s
I20240502 01:52:54.144131 11898 tablet_bootstrap.cc:492] T f7042550f3fe4a4780566f2fbdda3678 P 7b7a47245f294ce9a264fcc8f6351c0f: Bootstrap starting.
I20240502 01:52:54.144774 11896 raft_consensus.cc:348] T d4384df5dd744d399034ef24e55fbbf9 P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } }
I20240502 01:52:54.145026 11896 raft_consensus.cc:374] T d4384df5dd744d399034ef24e55fbbf9 P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:54.145131 11896 raft_consensus.cc:729] T d4384df5dd744d399034ef24e55fbbf9 P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a885359a82bd4515913f7b273591f851, State: Initialized, Role: FOLLOWER
I20240502 01:52:54.145403 11896 consensus_queue.cc:260] T d4384df5dd744d399034ef24e55fbbf9 P a885359a82bd4515913f7b273591f851 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } }
I20240502 01:52:54.146080 11896 ts_tablet_manager.cc:1398] T d4384df5dd744d399034ef24e55fbbf9 P a885359a82bd4515913f7b273591f851: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 01:52:54.148269 11900 tablet_bootstrap.cc:492] T d4384df5dd744d399034ef24e55fbbf9 P ede1dd598e3541138a19a3ea5c052bc9: No bootstrap required, opened a new log
I20240502 01:52:54.149924 11900 ts_tablet_manager.cc:1367] T d4384df5dd744d399034ef24e55fbbf9 P ede1dd598e3541138a19a3ea5c052bc9: Time spent bootstrapping tablet: real 0.023s	user 0.003s	sys 0.007s
I20240502 01:52:54.151037 11900 raft_consensus.cc:348] T d4384df5dd744d399034ef24e55fbbf9 P ede1dd598e3541138a19a3ea5c052bc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } }
I20240502 01:52:54.151371 11900 raft_consensus.cc:374] T d4384df5dd744d399034ef24e55fbbf9 P ede1dd598e3541138a19a3ea5c052bc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:54.151407 11897 tablet_bootstrap.cc:492] T d4384df5dd744d399034ef24e55fbbf9 P a8c4a518929d4bb3a88bdfb1649493bd: No bootstrap required, opened a new log
I20240502 01:52:54.151546 11900 raft_consensus.cc:729] T d4384df5dd744d399034ef24e55fbbf9 P ede1dd598e3541138a19a3ea5c052bc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ede1dd598e3541138a19a3ea5c052bc9, State: Initialized, Role: FOLLOWER
I20240502 01:52:54.151952 11897 ts_tablet_manager.cc:1367] T d4384df5dd744d399034ef24e55fbbf9 P a8c4a518929d4bb3a88bdfb1649493bd: Time spent bootstrapping tablet: real 0.030s	user 0.006s	sys 0.000s
I20240502 01:52:54.152674 11900 consensus_queue.cc:260] T d4384df5dd744d399034ef24e55fbbf9 P ede1dd598e3541138a19a3ea5c052bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } }
I20240502 01:52:54.153026 11897 raft_consensus.cc:348] T d4384df5dd744d399034ef24e55fbbf9 P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } }
I20240502 01:52:54.153316 11897 raft_consensus.cc:374] T d4384df5dd744d399034ef24e55fbbf9 P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:54.153438 11897 raft_consensus.cc:729] T d4384df5dd744d399034ef24e55fbbf9 P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8c4a518929d4bb3a88bdfb1649493bd, State: Initialized, Role: FOLLOWER
I20240502 01:52:54.153743 11897 consensus_queue.cc:260] T d4384df5dd744d399034ef24e55fbbf9 P a8c4a518929d4bb3a88bdfb1649493bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } }
I20240502 01:52:54.154371 11897 ts_tablet_manager.cc:1398] T d4384df5dd744d399034ef24e55fbbf9 P a8c4a518929d4bb3a88bdfb1649493bd: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240502 01:52:54.154632 11895 tablet_bootstrap.cc:654] T e71b5c9725d9400d8ed9faad7276a474 P 34da849acb2045a9a2ef03933592dd08: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:54.155373 11900 ts_tablet_manager.cc:1398] T d4384df5dd744d399034ef24e55fbbf9 P ede1dd598e3541138a19a3ea5c052bc9: Time spent starting tablet: real 0.005s	user 0.000s	sys 0.002s
I20240502 01:52:54.156486 11899 tablet_bootstrap.cc:492] T e71b5c9725d9400d8ed9faad7276a474 P fd95bcd4cdcb4cc5966050ffe2cffdff: No bootstrap required, opened a new log
I20240502 01:52:54.156675 11899 ts_tablet_manager.cc:1367] T e71b5c9725d9400d8ed9faad7276a474 P fd95bcd4cdcb4cc5966050ffe2cffdff: Time spent bootstrapping tablet: real 0.031s	user 0.000s	sys 0.009s
I20240502 01:52:54.157655 11899 raft_consensus.cc:348] T e71b5c9725d9400d8ed9faad7276a474 P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } }
I20240502 01:52:54.157886 11899 raft_consensus.cc:374] T e71b5c9725d9400d8ed9faad7276a474 P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:54.150488 11898 tablet_bootstrap.cc:654] T f7042550f3fe4a4780566f2fbdda3678 P 7b7a47245f294ce9a264fcc8f6351c0f: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:54.159166 11899 raft_consensus.cc:729] T e71b5c9725d9400d8ed9faad7276a474 P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd95bcd4cdcb4cc5966050ffe2cffdff, State: Initialized, Role: FOLLOWER
I20240502 01:52:54.159627 11899 consensus_queue.cc:260] T e71b5c9725d9400d8ed9faad7276a474 P fd95bcd4cdcb4cc5966050ffe2cffdff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } }
I20240502 01:52:54.161598 11899 ts_tablet_manager.cc:1398] T e71b5c9725d9400d8ed9faad7276a474 P fd95bcd4cdcb4cc5966050ffe2cffdff: Time spent starting tablet: real 0.005s	user 0.003s	sys 0.000s
I20240502 01:52:54.161965 11899 tablet_bootstrap.cc:492] T f7042550f3fe4a4780566f2fbdda3678 P fd95bcd4cdcb4cc5966050ffe2cffdff: Bootstrap starting.
I20240502 01:52:54.163831 11899 tablet_bootstrap.cc:654] T f7042550f3fe4a4780566f2fbdda3678 P fd95bcd4cdcb4cc5966050ffe2cffdff: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:54.171694 11898 tablet_bootstrap.cc:492] T f7042550f3fe4a4780566f2fbdda3678 P 7b7a47245f294ce9a264fcc8f6351c0f: No bootstrap required, opened a new log
I20240502 01:52:54.171907 11898 ts_tablet_manager.cc:1367] T f7042550f3fe4a4780566f2fbdda3678 P 7b7a47245f294ce9a264fcc8f6351c0f: Time spent bootstrapping tablet: real 0.028s	user 0.005s	sys 0.000s
I20240502 01:52:54.172123 11895 tablet_bootstrap.cc:492] T e71b5c9725d9400d8ed9faad7276a474 P 34da849acb2045a9a2ef03933592dd08: No bootstrap required, opened a new log
I20240502 01:52:54.172269 11899 tablet_bootstrap.cc:492] T f7042550f3fe4a4780566f2fbdda3678 P fd95bcd4cdcb4cc5966050ffe2cffdff: No bootstrap required, opened a new log
I20240502 01:52:54.172312 11895 ts_tablet_manager.cc:1367] T e71b5c9725d9400d8ed9faad7276a474 P 34da849acb2045a9a2ef03933592dd08: Time spent bootstrapping tablet: real 0.039s	user 0.001s	sys 0.006s
I20240502 01:52:54.172463 11899 ts_tablet_manager.cc:1367] T f7042550f3fe4a4780566f2fbdda3678 P fd95bcd4cdcb4cc5966050ffe2cffdff: Time spent bootstrapping tablet: real 0.011s	user 0.005s	sys 0.004s
I20240502 01:52:54.172806 11898 raft_consensus.cc:348] T f7042550f3fe4a4780566f2fbdda3678 P 7b7a47245f294ce9a264fcc8f6351c0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } }
I20240502 01:52:54.173020 11898 raft_consensus.cc:374] T f7042550f3fe4a4780566f2fbdda3678 P 7b7a47245f294ce9a264fcc8f6351c0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:54.173136 11898 raft_consensus.cc:729] T f7042550f3fe4a4780566f2fbdda3678 P 7b7a47245f294ce9a264fcc8f6351c0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b7a47245f294ce9a264fcc8f6351c0f, State: Initialized, Role: FOLLOWER
I20240502 01:52:54.173202 11895 raft_consensus.cc:348] T e71b5c9725d9400d8ed9faad7276a474 P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } }
I20240502 01:52:54.173306 11899 raft_consensus.cc:348] T f7042550f3fe4a4780566f2fbdda3678 P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } }
I20240502 01:52:54.173489 11895 raft_consensus.cc:374] T e71b5c9725d9400d8ed9faad7276a474 P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:54.173547 11899 raft_consensus.cc:374] T f7042550f3fe4a4780566f2fbdda3678 P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:54.173606 11895 raft_consensus.cc:729] T e71b5c9725d9400d8ed9faad7276a474 P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34da849acb2045a9a2ef03933592dd08, State: Initialized, Role: FOLLOWER
I20240502 01:52:54.173632 11899 raft_consensus.cc:729] T f7042550f3fe4a4780566f2fbdda3678 P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd95bcd4cdcb4cc5966050ffe2cffdff, State: Initialized, Role: FOLLOWER
I20240502 01:52:54.173947 11899 consensus_queue.cc:260] T f7042550f3fe4a4780566f2fbdda3678 P fd95bcd4cdcb4cc5966050ffe2cffdff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } }
I20240502 01:52:54.173947 11895 consensus_queue.cc:260] T e71b5c9725d9400d8ed9faad7276a474 P 34da849acb2045a9a2ef03933592dd08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } }
I20240502 01:52:54.174702 11899 ts_tablet_manager.cc:1398] T f7042550f3fe4a4780566f2fbdda3678 P fd95bcd4cdcb4cc5966050ffe2cffdff: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.002s
I20240502 01:52:54.174729 11895 ts_tablet_manager.cc:1398] T e71b5c9725d9400d8ed9faad7276a474 P 34da849acb2045a9a2ef03933592dd08: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240502 01:52:54.173817 11898 consensus_queue.cc:260] T f7042550f3fe4a4780566f2fbdda3678 P 7b7a47245f294ce9a264fcc8f6351c0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } }
I20240502 01:52:54.176193 11898 ts_tablet_manager.cc:1398] T f7042550f3fe4a4780566f2fbdda3678 P 7b7a47245f294ce9a264fcc8f6351c0f: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.000s
I20240502 01:52:54.176643 11895 tablet_bootstrap.cc:492] T f7042550f3fe4a4780566f2fbdda3678 P 34da849acb2045a9a2ef03933592dd08: Bootstrap starting.
I20240502 01:52:54.178532 11895 tablet_bootstrap.cc:654] T f7042550f3fe4a4780566f2fbdda3678 P 34da849acb2045a9a2ef03933592dd08: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:54.182853 11895 tablet_bootstrap.cc:492] T f7042550f3fe4a4780566f2fbdda3678 P 34da849acb2045a9a2ef03933592dd08: No bootstrap required, opened a new log
I20240502 01:52:54.183063 11895 ts_tablet_manager.cc:1367] T f7042550f3fe4a4780566f2fbdda3678 P 34da849acb2045a9a2ef03933592dd08: Time spent bootstrapping tablet: real 0.006s	user 0.000s	sys 0.005s
I20240502 01:52:54.183933 11895 raft_consensus.cc:348] T f7042550f3fe4a4780566f2fbdda3678 P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } }
I20240502 01:52:54.184201 11895 raft_consensus.cc:374] T f7042550f3fe4a4780566f2fbdda3678 P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:54.184412 11895 raft_consensus.cc:729] T f7042550f3fe4a4780566f2fbdda3678 P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34da849acb2045a9a2ef03933592dd08, State: Initialized, Role: FOLLOWER
I20240502 01:52:54.184751 11895 consensus_queue.cc:260] T f7042550f3fe4a4780566f2fbdda3678 P 34da849acb2045a9a2ef03933592dd08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } }
I20240502 01:52:54.185350 11895 ts_tablet_manager.cc:1398] T f7042550f3fe4a4780566f2fbdda3678 P 34da849acb2045a9a2ef03933592dd08: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 01:52:54.224581 11857 raft_consensus.cc:482] T d4384df5dd744d399034ef24e55fbbf9 P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:54.224812 11857 raft_consensus.cc:504] T d4384df5dd744d399034ef24e55fbbf9 P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } }
I20240502 01:52:54.225891 11857 leader_election.cc:288] T d4384df5dd744d399034ef24e55fbbf9 P a885359a82bd4515913f7b273591f851 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a8c4a518929d4bb3a88bdfb1649493bd (127.9.176.197:44061), ede1dd598e3541138a19a3ea5c052bc9 (127.9.176.195:38701)
I20240502 01:52:54.226382 10644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4384df5dd744d399034ef24e55fbbf9" candidate_uuid: "a885359a82bd4515913f7b273591f851" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" is_pre_election: true
I20240502 01:52:54.226476 10376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4384df5dd744d399034ef24e55fbbf9" candidate_uuid: "a885359a82bd4515913f7b273591f851" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ede1dd598e3541138a19a3ea5c052bc9" is_pre_election: true
I20240502 01:52:54.226666 10644 raft_consensus.cc:2454] T d4384df5dd744d399034ef24e55fbbf9 P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a885359a82bd4515913f7b273591f851 in term 0.
I20240502 01:52:54.226749 10376 raft_consensus.cc:2454] T d4384df5dd744d399034ef24e55fbbf9 P ede1dd598e3541138a19a3ea5c052bc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a885359a82bd4515913f7b273591f851 in term 0.
I20240502 01:52:54.227180 10046 leader_election.cc:302] T d4384df5dd744d399034ef24e55fbbf9 P a885359a82bd4515913f7b273591f851 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a885359a82bd4515913f7b273591f851, a8c4a518929d4bb3a88bdfb1649493bd; no voters: 
I20240502 01:52:54.227469 11857 raft_consensus.cc:2781] T d4384df5dd744d399034ef24e55fbbf9 P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:54.227630 11857 raft_consensus.cc:482] T d4384df5dd744d399034ef24e55fbbf9 P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:54.227751 11857 raft_consensus.cc:3037] T d4384df5dd744d399034ef24e55fbbf9 P a885359a82bd4515913f7b273591f851 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:54.229941 11857 raft_consensus.cc:504] T d4384df5dd744d399034ef24e55fbbf9 P a885359a82bd4515913f7b273591f851 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } }
I20240502 01:52:54.230489 11857 leader_election.cc:288] T d4384df5dd744d399034ef24e55fbbf9 P a885359a82bd4515913f7b273591f851 [CANDIDATE]: Term 1 election: Requested vote from peers a8c4a518929d4bb3a88bdfb1649493bd (127.9.176.197:44061), ede1dd598e3541138a19a3ea5c052bc9 (127.9.176.195:38701)
I20240502 01:52:54.230993 10644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4384df5dd744d399034ef24e55fbbf9" candidate_uuid: "a885359a82bd4515913f7b273591f851" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8c4a518929d4bb3a88bdfb1649493bd"
I20240502 01:52:54.231009 10376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4384df5dd744d399034ef24e55fbbf9" candidate_uuid: "a885359a82bd4515913f7b273591f851" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ede1dd598e3541138a19a3ea5c052bc9"
I20240502 01:52:54.231216 10644 raft_consensus.cc:3037] T d4384df5dd744d399034ef24e55fbbf9 P a8c4a518929d4bb3a88bdfb1649493bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:54.231256 10376 raft_consensus.cc:3037] T d4384df5dd744d399034ef24e55fbbf9 P ede1dd598e3541138a19a3ea5c052bc9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:54.232012 11856 raft_consensus.cc:482] T e71b5c9725d9400d8ed9faad7276a474 P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:54.232218 11856 raft_consensus.cc:504] T e71b5c9725d9400d8ed9faad7276a474 P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } }
I20240502 01:52:54.233027 11856 leader_election.cc:288] T e71b5c9725d9400d8ed9faad7276a474 P 34da849acb2045a9a2ef03933592dd08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fd95bcd4cdcb4cc5966050ffe2cffdff (127.9.176.198:33039), 7b7a47245f294ce9a264fcc8f6351c0f (127.9.176.194:43645)
I20240502 01:52:54.233547 10644 raft_consensus.cc:2454] T d4384df5dd744d399034ef24e55fbbf9 P a8c4a518929d4bb3a88bdfb1649493bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a885359a82bd4515913f7b273591f851 in term 1.
I20240502 01:52:54.233508 10779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e71b5c9725d9400d8ed9faad7276a474" candidate_uuid: "34da849acb2045a9a2ef03933592dd08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" is_pre_election: true
I20240502 01:52:54.233769 10779 raft_consensus.cc:2454] T e71b5c9725d9400d8ed9faad7276a474 P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34da849acb2045a9a2ef03933592dd08 in term 0.
I20240502 01:52:54.233846 10376 raft_consensus.cc:2454] T d4384df5dd744d399034ef24e55fbbf9 P ede1dd598e3541138a19a3ea5c052bc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a885359a82bd4515913f7b273591f851 in term 1.
I20240502 01:52:54.234306 10445 leader_election.cc:302] T e71b5c9725d9400d8ed9faad7276a474 P 34da849acb2045a9a2ef03933592dd08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34da849acb2045a9a2ef03933592dd08, fd95bcd4cdcb4cc5966050ffe2cffdff; no voters: 
I20240502 01:52:54.234660 11856 raft_consensus.cc:2781] T e71b5c9725d9400d8ed9faad7276a474 P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:54.234746 11856 raft_consensus.cc:482] T e71b5c9725d9400d8ed9faad7276a474 P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:54.234642 10244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e71b5c9725d9400d8ed9faad7276a474" candidate_uuid: "34da849acb2045a9a2ef03933592dd08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" is_pre_election: true
I20240502 01:52:54.234849 11856 raft_consensus.cc:3037] T e71b5c9725d9400d8ed9faad7276a474 P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:54.234835 10046 leader_election.cc:302] T d4384df5dd744d399034ef24e55fbbf9 P a885359a82bd4515913f7b273591f851 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: a885359a82bd4515913f7b273591f851, a8c4a518929d4bb3a88bdfb1649493bd, ede1dd598e3541138a19a3ea5c052bc9; no voters: 
I20240502 01:52:54.234962 10244 raft_consensus.cc:2454] T e71b5c9725d9400d8ed9faad7276a474 P 7b7a47245f294ce9a264fcc8f6351c0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34da849acb2045a9a2ef03933592dd08 in term 0.
I20240502 01:52:54.235201 11857 raft_consensus.cc:2781] T d4384df5dd744d399034ef24e55fbbf9 P a885359a82bd4515913f7b273591f851 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:54.235458 11857 raft_consensus.cc:686] T d4384df5dd744d399034ef24e55fbbf9 P a885359a82bd4515913f7b273591f851 [term 1 LEADER]: Becoming Leader. State: Replica: a885359a82bd4515913f7b273591f851, State: Running, Role: LEADER
I20240502 01:52:54.235826 11857 consensus_queue.cc:237] T d4384df5dd744d399034ef24e55fbbf9 P a885359a82bd4515913f7b273591f851 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } }
I20240502 01:52:54.237129 11856 raft_consensus.cc:504] T e71b5c9725d9400d8ed9faad7276a474 P 34da849acb2045a9a2ef03933592dd08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } }
I20240502 01:52:54.238723 10779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e71b5c9725d9400d8ed9faad7276a474" candidate_uuid: "34da849acb2045a9a2ef03933592dd08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff"
I20240502 01:52:54.238858 11856 leader_election.cc:288] T e71b5c9725d9400d8ed9faad7276a474 P 34da849acb2045a9a2ef03933592dd08 [CANDIDATE]: Term 1 election: Requested vote from peers fd95bcd4cdcb4cc5966050ffe2cffdff (127.9.176.198:33039), 7b7a47245f294ce9a264fcc8f6351c0f (127.9.176.194:43645)
I20240502 01:52:54.238971 10779 raft_consensus.cc:3037] T e71b5c9725d9400d8ed9faad7276a474 P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:54.239531 10244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e71b5c9725d9400d8ed9faad7276a474" candidate_uuid: "34da849acb2045a9a2ef03933592dd08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b7a47245f294ce9a264fcc8f6351c0f"
I20240502 01:52:54.239818 10244 raft_consensus.cc:3037] T e71b5c9725d9400d8ed9faad7276a474 P 7b7a47245f294ce9a264fcc8f6351c0f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:54.239553  9962 catalog_manager.cc:5484] T d4384df5dd744d399034ef24e55fbbf9 P a885359a82bd4515913f7b273591f851 reported cstate change: term changed from 0 to 1, leader changed from <none> to a885359a82bd4515913f7b273591f851 (127.9.176.193). New cstate: current_term: 1 leader_uuid: "a885359a82bd4515913f7b273591f851" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a885359a82bd4515913f7b273591f851" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38373 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 } health_report { overall_health: UNKNOWN } } }
I20240502 01:52:54.242126 10779 raft_consensus.cc:2454] T e71b5c9725d9400d8ed9faad7276a474 P fd95bcd4cdcb4cc5966050ffe2cffdff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34da849acb2045a9a2ef03933592dd08 in term 1.
I20240502 01:52:54.242723 10244 raft_consensus.cc:2454] T e71b5c9725d9400d8ed9faad7276a474 P 7b7a47245f294ce9a264fcc8f6351c0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34da849acb2045a9a2ef03933592dd08 in term 1.
I20240502 01:52:54.242826 10445 leader_election.cc:302] T e71b5c9725d9400d8ed9faad7276a474 P 34da849acb2045a9a2ef03933592dd08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34da849acb2045a9a2ef03933592dd08, fd95bcd4cdcb4cc5966050ffe2cffdff; no voters: 
I20240502 01:52:54.243253 11856 raft_consensus.cc:2781] T e71b5c9725d9400d8ed9faad7276a474 P 34da849acb2045a9a2ef03933592dd08 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:54.243535 11856 raft_consensus.cc:686] T e71b5c9725d9400d8ed9faad7276a474 P 34da849acb2045a9a2ef03933592dd08 [term 1 LEADER]: Becoming Leader. State: Replica: 34da849acb2045a9a2ef03933592dd08, State: Running, Role: LEADER
I20240502 01:52:54.243896 11856 consensus_queue.cc:237] T e71b5c9725d9400d8ed9faad7276a474 P 34da849acb2045a9a2ef03933592dd08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } }
I20240502 01:52:54.247241  9962 catalog_manager.cc:5484] T e71b5c9725d9400d8ed9faad7276a474 P 34da849acb2045a9a2ef03933592dd08 reported cstate change: term changed from 0 to 1, leader changed from <none> to 34da849acb2045a9a2ef03933592dd08 (127.9.176.196). New cstate: current_term: 1 leader_uuid: "34da849acb2045a9a2ef03933592dd08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } health_report { overall_health: HEALTHY } } }
I20240502 01:52:54.488238 11312 raft_consensus.cc:482] T f7042550f3fe4a4780566f2fbdda3678 P 7b7a47245f294ce9a264fcc8f6351c0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:54.488509 11312 raft_consensus.cc:504] T f7042550f3fe4a4780566f2fbdda3678 P 7b7a47245f294ce9a264fcc8f6351c0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } }
I20240502 01:52:54.489342 11312 leader_election.cc:288] T f7042550f3fe4a4780566f2fbdda3678 P 7b7a47245f294ce9a264fcc8f6351c0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fd95bcd4cdcb4cc5966050ffe2cffdff (127.9.176.198:33039), 34da849acb2045a9a2ef03933592dd08 (127.9.176.196:44521)
I20240502 01:52:54.489955 10778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7042550f3fe4a4780566f2fbdda3678" candidate_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" is_pre_election: true
I20240502 01:52:54.489953 10511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7042550f3fe4a4780566f2fbdda3678" candidate_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34da849acb2045a9a2ef03933592dd08" is_pre_election: true
I20240502 01:52:54.490209 10778 raft_consensus.cc:2454] T f7042550f3fe4a4780566f2fbdda3678 P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b7a47245f294ce9a264fcc8f6351c0f in term 0.
I20240502 01:52:54.490213 10511 raft_consensus.cc:2454] T f7042550f3fe4a4780566f2fbdda3678 P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b7a47245f294ce9a264fcc8f6351c0f in term 0.
I20240502 01:52:54.490697 10179 leader_election.cc:302] T f7042550f3fe4a4780566f2fbdda3678 P 7b7a47245f294ce9a264fcc8f6351c0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b7a47245f294ce9a264fcc8f6351c0f, fd95bcd4cdcb4cc5966050ffe2cffdff; no voters: 
I20240502 01:52:54.491081 11312 raft_consensus.cc:2781] T f7042550f3fe4a4780566f2fbdda3678 P 7b7a47245f294ce9a264fcc8f6351c0f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:54.491219 11312 raft_consensus.cc:482] T f7042550f3fe4a4780566f2fbdda3678 P 7b7a47245f294ce9a264fcc8f6351c0f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:54.491333 11312 raft_consensus.cc:3037] T f7042550f3fe4a4780566f2fbdda3678 P 7b7a47245f294ce9a264fcc8f6351c0f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:54.493454 11312 raft_consensus.cc:504] T f7042550f3fe4a4780566f2fbdda3678 P 7b7a47245f294ce9a264fcc8f6351c0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } }
I20240502 01:52:54.494159 11312 leader_election.cc:288] T f7042550f3fe4a4780566f2fbdda3678 P 7b7a47245f294ce9a264fcc8f6351c0f [CANDIDATE]: Term 1 election: Requested vote from peers fd95bcd4cdcb4cc5966050ffe2cffdff (127.9.176.198:33039), 34da849acb2045a9a2ef03933592dd08 (127.9.176.196:44521)
I20240502 01:52:54.494779 10511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7042550f3fe4a4780566f2fbdda3678" candidate_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34da849acb2045a9a2ef03933592dd08"
I20240502 01:52:54.494793 10778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7042550f3fe4a4780566f2fbdda3678" candidate_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff"
I20240502 01:52:54.495016 10778 raft_consensus.cc:3037] T f7042550f3fe4a4780566f2fbdda3678 P fd95bcd4cdcb4cc5966050ffe2cffdff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:54.495016 10511 raft_consensus.cc:3037] T f7042550f3fe4a4780566f2fbdda3678 P 34da849acb2045a9a2ef03933592dd08 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:54.497085 10511 raft_consensus.cc:2454] T f7042550f3fe4a4780566f2fbdda3678 P 34da849acb2045a9a2ef03933592dd08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b7a47245f294ce9a264fcc8f6351c0f in term 1.
I20240502 01:52:54.497256 10778 raft_consensus.cc:2454] T f7042550f3fe4a4780566f2fbdda3678 P fd95bcd4cdcb4cc5966050ffe2cffdff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b7a47245f294ce9a264fcc8f6351c0f in term 1.
I20240502 01:52:54.497689 10177 leader_election.cc:302] T f7042550f3fe4a4780566f2fbdda3678 P 7b7a47245f294ce9a264fcc8f6351c0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34da849acb2045a9a2ef03933592dd08, 7b7a47245f294ce9a264fcc8f6351c0f; no voters: 
I20240502 01:52:54.498044 11312 raft_consensus.cc:2781] T f7042550f3fe4a4780566f2fbdda3678 P 7b7a47245f294ce9a264fcc8f6351c0f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:54.498430 11312 raft_consensus.cc:686] T f7042550f3fe4a4780566f2fbdda3678 P 7b7a47245f294ce9a264fcc8f6351c0f [term 1 LEADER]: Becoming Leader. State: Replica: 7b7a47245f294ce9a264fcc8f6351c0f, State: Running, Role: LEADER
I20240502 01:52:54.498768 11312 consensus_queue.cc:237] T f7042550f3fe4a4780566f2fbdda3678 P 7b7a47245f294ce9a264fcc8f6351c0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } }
I20240502 01:52:54.502029  9962 catalog_manager.cc:5484] T f7042550f3fe4a4780566f2fbdda3678 P 7b7a47245f294ce9a264fcc8f6351c0f reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b7a47245f294ce9a264fcc8f6351c0f (127.9.176.194). New cstate: current_term: 1 leader_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 } health_report { overall_health: HEALTHY } } }
I20240502 01:52:54.659741 11857 consensus_queue.cc:1035] T d4384df5dd744d399034ef24e55fbbf9 P a885359a82bd4515913f7b273591f851 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8c4a518929d4bb3a88bdfb1649493bd" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 44061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:54.684079 11857 consensus_queue.cc:1035] T d4384df5dd744d399034ef24e55fbbf9 P a885359a82bd4515913f7b273591f851 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ede1dd598e3541138a19a3ea5c052bc9" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:54.734521 11851 consensus_queue.cc:1035] T e71b5c9725d9400d8ed9faad7276a474 P 34da849acb2045a9a2ef03933592dd08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:54.739975 11856 consensus_queue.cc:1035] T e71b5c9725d9400d8ed9faad7276a474 P 34da849acb2045a9a2ef03933592dd08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b7a47245f294ce9a264fcc8f6351c0f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:52:54.961126 11914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:54.961601 11914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:54.970431 11914 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_11'
I20240502 01:52:55.055209 11312 consensus_queue.cc:1035] T f7042550f3fe4a4780566f2fbdda3678 P 7b7a47245f294ce9a264fcc8f6351c0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd95bcd4cdcb4cc5966050ffe2cffdff" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 33039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:55.059505 11312 consensus_queue.cc:1035] T f7042550f3fe4a4780566f2fbdda3678 P 7b7a47245f294ce9a264fcc8f6351c0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "34da849acb2045a9a2ef03933592dd08" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 44521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 320.389 ms
  time per row: 0.160195 ms
-----------------------------------------------------
Suppressions used:
  count      bytes template
      2        596 libcrypto.so.1.1
-----------------------------------------------------

I20240502 01:52:57.074205 10492 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:52:57.078126 10091 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:52:57.082098 10225 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:52:57.111790 10358 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:52:57.126307 10623 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:52:57.158531 10759 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:52:57.391031 10492 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:52:57.409865 10623 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:52:57.410967 10358 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:52:57.417739 10225 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:52:57.428556 10759 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:52:57.434872 10091 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:52:57.731005 10091 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:52:57.733963 10492 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:52:57.749249 10358 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:52:57.753439 10623 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:52:57.771135 10225 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:52:57.771625 10759 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:52:58.045816 10492 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:52:58.051671 10623 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:52:58.052100 10225 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:52:58.059187 10091 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:52:58.068042 10358 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:52:58.089040 10759 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:52:58.411268 10492 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:52:58.411819 10225 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:52:58.433434 10358 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:52:58.441581 10623 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:52:58.448166 10759 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:52:58.451480 10091 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:52:58.898907 10623 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:52:58.905514 10492 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:52:58.911373 10091 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:52:58.911818 10759 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:52:58.915596 10358 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:52:58.924325 10225 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
               UUID               |       Address       | Status
----------------------------------+---------------------+---------
 7684868bab274f87b883d33cfb6b8214 | 127.9.176.254:36089 | HEALTHY

Unusual flags for Master:
                   Flag                   |                                                                                    Value                                                                                     |         Tags          |         Master
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
 ipki_ca_key_size                         | 768                                                                                                                                                                          | experimental          | all 1 server(s) checked
 ipki_server_key_size                     | 768                                                                                                                                                                          | experimental          | all 1 server(s) checked
 never_fsync                              | true                                                                                                                                                                         | unsafe,advanced       | all 1 server(s) checked
 openssl_security_level_override          | 0                                                                                                                                                                            | unsafe,hidden         | all 1 server(s) checked
 raft_prepare_replacement_before_eviction | true                                                                                                                                                                         | experimental,advanced | all 1 server(s) checked
 rpc_reuseport                            | true                                                                                                                                                                         | experimental          | all 1 server(s) checked
 rpc_server_allow_ephemeral_ports         | true                                                                                                                                                                         | unsafe                | all 1 server(s) checked
 server_dump_info_format                  | pb                                                                                                                                                                           | hidden                | all 1 server(s) checked
 server_dump_info_path                    | /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden                | all 1 server(s) checked
 tsk_num_rsa_bits                         | 512                                                                                                                                                                          | experimental          | all 1 server(s) checked

Flags of checked categories for Master:
        Flag         |        Value        |         Master
---------------------+---------------------+-------------------------
 builtin_ntp_servers | 127.9.176.212:38509 | all 1 server(s) checked
 time_source         | builtin             | all 1 server(s) checked

Tablet Server Summary
               UUID               |       Address       | Status  | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
 34da849acb2045a9a2ef03933592dd08 | 127.9.176.196:44521 | HEALTHY | /B       |       7        |       0
 7b7a47245f294ce9a264fcc8f6351c0f | 127.9.176.194:43645 | HEALTHY | /A       |       2        |       0
 a885359a82bd4515913f7b273591f851 | 127.9.176.193:38373 | HEALTHY | /A       |      10        |       0
 a8c4a518929d4bb3a88bdfb1649493bd | 127.9.176.197:44061 | HEALTHY | /C       |       6        |       0
 ede1dd598e3541138a19a3ea5c052bc9 | 127.9.176.195:38701 | HEALTHY | /B       |       2        |       0
 fd95bcd4cdcb4cc5966050ffe2cffdff | 127.9.176.198:33039 | HEALTHY | /C       |       9        |       0

Tablet Server Location Summary
 Location |  Count
----------+---------
 /C       |       2
 /B       |       2
 /A       |       2

Unusual flags for Tablet Server:
                   Flag                   |                                                                                  Value                                                                                   |         Tags          |      Tablet Server
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
 ipki_server_key_size                     | 768                                                                                                                                                                      | experimental          | all 6 server(s) checked
 local_ip_for_outbound_sockets            | 127.9.176.193                                                                                                                                                            | experimental          | 127.9.176.193:38373
 local_ip_for_outbound_sockets            | 127.9.176.194                                                                                                                                                            | experimental          | 127.9.176.194:43645
 local_ip_for_outbound_sockets            | 127.9.176.195                                                                                                                                                            | experimental          | 127.9.176.195:38701
 local_ip_for_outbound_sockets            | 127.9.176.196                                                                                                                                                            | experimental          | 127.9.176.196:44521
 local_ip_for_outbound_sockets            | 127.9.176.197                                                                                                                                                            | experimental          | 127.9.176.197:44061
 local_ip_for_outbound_sockets            | 127.9.176.198                                                                                                                                                            | experimental          | 127.9.176.198:33039
 never_fsync                              | true                                                                                                                                                                     | unsafe,advanced       | all 6 server(s) checked
 openssl_security_level_override          | 0                                                                                                                                                                        | unsafe,hidden         | all 6 server(s) checked
 raft_prepare_replacement_before_eviction | true                                                                                                                                                                     | experimental,advanced | all 6 server(s) checked
 rpc_server_allow_ephemeral_ports         | true                                                                                                                                                                     | unsafe                | all 6 server(s) checked
 server_dump_info_format                  | pb                                                                                                                                                                       | hidden                | all 6 server(s) checked
 server_dump_info_path                    | /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden                | 127.9.176.193:38373
 server_dump_info_path                    | /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden                | 127.9.176.194:43645
 server_dump_info_path                    | /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden                | 127.9.176.195:38701
 server_dump_info_path                    | /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden                | 127.9.176.196:44521
 server_dump_info_path                    | /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden                | 127.9.176.197:44061
 server_dump_info_path                    | /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden                | 127.9.176.198:33039

Flags of checked categories for Tablet Server:
        Flag         |        Value        |      Tablet Server
---------------------+---------------------+-------------------------
 builtin_ntp_servers | 127.9.176.212:38509 | all 6 server(s) checked
 time_source         | builtin             | all 6 server(s) checked

Version Summary
     Version     |         Servers
-----------------+-------------------------
 1.18.0-SNAPSHOT | all 7 server(s) checked

Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
          Name           | RF | Status  | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-------------------------+----+---------+---------------+---------+------------+------------------+-------------
 rebalance_test_table_0  | 3  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_1  | 3  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_10 | 3  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_11 | 3  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_2  | 3  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_3  | 3  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_4  | 3  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_5  | 3  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_6  | 3  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_7  | 3  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_8  | 3  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_9  | 3  | HEALTHY | 3             | 3       | 0          | 0                | 0

Tablet Replica Count Summary
   Statistic    | Replica Count
----------------+---------------
 Minimum        | 18
 First Quartile | 18
 Median         | 18
 Third Quartile | 18
 Maximum        | 18

Total Count Summary
                | Total Count
----------------+-------------
 Masters        | 1
 Tablet Servers | 6
 Tables         | 12
 Tablets        | 36
 Replicas       | 108

==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set

OK
I20240502 01:52:59.017997  9923 log_verifier.cc:126] Checking tablet 09a892cf118d41d4bfffc1576731d54c
I20240502 01:52:59.059921  9923 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 09a892cf118d41d4bfffc1576731d54c
I20240502 01:52:59.060164  9923 log_verifier.cc:126] Checking tablet 1da6b311a749427f9fdd394bec7a033f
I20240502 01:52:59.097301  9923 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 1da6b311a749427f9fdd394bec7a033f
I20240502 01:52:59.097420  9923 log_verifier.cc:126] Checking tablet 209e5d85bfd543b19c0c55b657bf7931
I20240502 01:52:59.136512  9923 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 209e5d85bfd543b19c0c55b657bf7931
I20240502 01:52:59.136648  9923 log_verifier.cc:126] Checking tablet 2f803cd8e5bd420c8f0417d92af153b5
I20240502 01:52:59.188836  9923 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 2f803cd8e5bd420c8f0417d92af153b5
I20240502 01:52:59.188985  9923 log_verifier.cc:126] Checking tablet 3a4f610cdc9d4dd48fd44924e406d929
I20240502 01:52:59.244144  9923 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 3a4f610cdc9d4dd48fd44924e406d929
I20240502 01:52:59.244272  9923 log_verifier.cc:126] Checking tablet 4c79cbf474df4ff0974e5611fbba66cd
I20240502 01:52:59.282328  9923 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 4c79cbf474df4ff0974e5611fbba66cd
I20240502 01:52:59.282459  9923 log_verifier.cc:126] Checking tablet 6488dcdd5ab04e35bbd720f4985ceb41
I20240502 01:52:59.321681  9923 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 6488dcdd5ab04e35bbd720f4985ceb41
I20240502 01:52:59.321797  9923 log_verifier.cc:126] Checking tablet 673fc226d6b441219d1153741b914808
I20240502 01:52:59.383175  9923 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 673fc226d6b441219d1153741b914808
I20240502 01:52:59.383294  9923 log_verifier.cc:126] Checking tablet 75d03e64bee04a2092364ca91241cf4f
I20240502 01:52:59.427070  9923 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 75d03e64bee04a2092364ca91241cf4f
I20240502 01:52:59.427194  9923 log_verifier.cc:126] Checking tablet 816e798bf21641c1872dc88d2fd3e820
I20240502 01:52:59.466666  9923 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 816e798bf21641c1872dc88d2fd3e820
I20240502 01:52:59.466789  9923 log_verifier.cc:126] Checking tablet 8fcb99ef99114ca59130ccef6d0803b3
I20240502 01:52:59.525336  9923 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 8fcb99ef99114ca59130ccef6d0803b3
I20240502 01:52:59.525461  9923 log_verifier.cc:126] Checking tablet 930071b72771448d98cbe659ee23c63a
I20240502 01:52:59.580535  9923 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 930071b72771448d98cbe659ee23c63a
I20240502 01:52:59.580657  9923 log_verifier.cc:126] Checking tablet 941430c4941d4ce7bf4252f0b56abecc
I20240502 01:52:59.634389  9923 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 941430c4941d4ce7bf4252f0b56abecc
I20240502 01:52:59.634554  9923 log_verifier.cc:126] Checking tablet 968bb8eb10d44a29b33299898ebe453c
I20240502 01:52:59.687945  9923 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 968bb8eb10d44a29b33299898ebe453c
I20240502 01:52:59.688112  9923 log_verifier.cc:126] Checking tablet a92d5a681b364e50896bdda0d92bb10a
I20240502 01:52:59.745222  9923 log_verifier.cc:177] Verified matching terms for 3 ops in tablet a92d5a681b364e50896bdda0d92bb10a
I20240502 01:52:59.745355  9923 log_verifier.cc:126] Checking tablet a97bdeed6d00481d8a352a67da51c7f6
I20240502 01:52:59.796602  9923 log_verifier.cc:177] Verified matching terms for 3 ops in tablet a97bdeed6d00481d8a352a67da51c7f6
I20240502 01:52:59.796792  9923 log_verifier.cc:126] Checking tablet ad11d50337ab43fdbc110dc200035fd1
I20240502 01:52:59.869606  9923 log_verifier.cc:177] Verified matching terms for 3 ops in tablet ad11d50337ab43fdbc110dc200035fd1
I20240502 01:52:59.869732  9923 log_verifier.cc:126] Checking tablet b0522638f70b4cb28359b7233b4b6121
I20240502 01:52:59.923710  9923 log_verifier.cc:177] Verified matching terms for 3 ops in tablet b0522638f70b4cb28359b7233b4b6121
I20240502 01:52:59.923826  9923 log_verifier.cc:126] Checking tablet b7218b5b2e2c4e1a862705edd8574f6b
I20240502 01:52:59.972003  9923 log_verifier.cc:177] Verified matching terms for 3 ops in tablet b7218b5b2e2c4e1a862705edd8574f6b
I20240502 01:52:59.972151  9923 log_verifier.cc:126] Checking tablet c264070710e8405ab9c2b3fe2cea46ae
I20240502 01:53:00.019868  9923 log_verifier.cc:177] Verified matching terms for 3 ops in tablet c264070710e8405ab9c2b3fe2cea46ae
I20240502 01:53:00.020030  9923 log_verifier.cc:126] Checking tablet c2730c626fe34084ab71d463dd2b8656
I20240502 01:53:00.070654  9923 log_verifier.cc:177] Verified matching terms for 3 ops in tablet c2730c626fe34084ab71d463dd2b8656
I20240502 01:53:00.070899  9923 log_verifier.cc:126] Checking tablet c96831f69a9b452c811152c9b0de7588
I20240502 01:53:00.115959  9923 log_verifier.cc:177] Verified matching terms for 3 ops in tablet c96831f69a9b452c811152c9b0de7588
I20240502 01:53:00.116097  9923 log_verifier.cc:126] Checking tablet c9f395cabd05425484aad0f82b0a3ca9
I20240502 01:53:00.169697  9923 log_verifier.cc:177] Verified matching terms for 3 ops in tablet c9f395cabd05425484aad0f82b0a3ca9
I20240502 01:53:00.169867  9923 log_verifier.cc:126] Checking tablet d31a3838cfbc4f3b9990048fbd340421
I20240502 01:53:00.223502  9923 log_verifier.cc:177] Verified matching terms for 3 ops in tablet d31a3838cfbc4f3b9990048fbd340421
I20240502 01:53:00.223682  9923 log_verifier.cc:126] Checking tablet d4384df5dd744d399034ef24e55fbbf9
I20240502 01:53:00.280707  9923 log_verifier.cc:177] Verified matching terms for 3 ops in tablet d4384df5dd744d399034ef24e55fbbf9
I20240502 01:53:00.280848  9923 log_verifier.cc:126] Checking tablet d8ffb869041f41259d08443b084ee9c0
I20240502 01:53:00.335031  9923 log_verifier.cc:177] Verified matching terms for 3 ops in tablet d8ffb869041f41259d08443b084ee9c0
I20240502 01:53:00.335161  9923 log_verifier.cc:126] Checking tablet de6e1081d71641248b0657bbdb824af8
I20240502 01:53:00.394492  9923 log_verifier.cc:177] Verified matching terms for 3 ops in tablet de6e1081d71641248b0657bbdb824af8
I20240502 01:53:00.394632  9923 log_verifier.cc:126] Checking tablet dea628b438e74180a688e5dca1f439ab
I20240502 01:53:00.445794  9923 log_verifier.cc:177] Verified matching terms for 3 ops in tablet dea628b438e74180a688e5dca1f439ab
I20240502 01:53:00.445940  9923 log_verifier.cc:126] Checking tablet e0f8668f24034001bebe8edabdf7ee9e
I20240502 01:53:00.503244  9923 log_verifier.cc:177] Verified matching terms for 3 ops in tablet e0f8668f24034001bebe8edabdf7ee9e
I20240502 01:53:00.503420  9923 log_verifier.cc:126] Checking tablet e185fe1a710b47aca0df91cb77bf60e0
I20240502 01:53:00.565866  9923 log_verifier.cc:177] Verified matching terms for 3 ops in tablet e185fe1a710b47aca0df91cb77bf60e0
I20240502 01:53:00.566017  9923 log_verifier.cc:126] Checking tablet e190679362c847c58c31c6bbfc293b32
I20240502 01:53:00.625985  9923 log_verifier.cc:177] Verified matching terms for 3 ops in tablet e190679362c847c58c31c6bbfc293b32
I20240502 01:53:00.626215  9923 log_verifier.cc:126] Checking tablet e71b5c9725d9400d8ed9faad7276a474
I20240502 01:53:00.689811  9923 log_verifier.cc:177] Verified matching terms for 3 ops in tablet e71b5c9725d9400d8ed9faad7276a474
I20240502 01:53:00.689970  9923 log_verifier.cc:126] Checking tablet eaefd6b8be0947da8d8cb87644f91c70
I20240502 01:53:00.773774  9923 log_verifier.cc:177] Verified matching terms for 3 ops in tablet eaefd6b8be0947da8d8cb87644f91c70
I20240502 01:53:00.773972  9923 log_verifier.cc:126] Checking tablet ec9d18ab22e1466c8daa6b554ccaeebb
I20240502 01:53:00.860628  9923 log_verifier.cc:177] Verified matching terms for 3 ops in tablet ec9d18ab22e1466c8daa6b554ccaeebb
I20240502 01:53:00.860796  9923 log_verifier.cc:126] Checking tablet f7042550f3fe4a4780566f2fbdda3678
I20240502 01:53:00.943645  9923 log_verifier.cc:177] Verified matching terms for 3 ops in tablet f7042550f3fe4a4780566f2fbdda3678
I20240502 01:53:00.943881  9923 log_verifier.cc:126] Checking tablet f72ee833c1d54106802dd9191e142284
I20240502 01:53:01.046828  9923 log_verifier.cc:177] Verified matching terms for 3 ops in tablet f72ee833c1d54106802dd9191e142284
I20240502 01:53:01.069940  9923 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu pid 10028
I20240502 01:53:01.077884 10151 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:53:01.228812  9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu with pid 10028
W20240502 01:53:01.254572 10580 connection.cc:535] server connection from 127.9.176.193:44559 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:53:01.254817 10580 connection.cc:535] client connection to 127.9.176.193:38373 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:53:01.255035 10580 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 01:53:01.255227  9923 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu pid 10161
W20240502 01:53:01.259912 10580 consensus_peers.cc:485] T b7218b5b2e2c4e1a862705edd8574f6b P a8c4a518929d4bb3a88bdfb1649493bd -> Peer a885359a82bd4515913f7b273591f851 (127.9.176.193:38373): Couldn't send request to peer a885359a82bd4515913f7b273591f851. Status: Network error: Client connection negotiation failed: client connection to 127.9.176.193:38373: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:53:01.260174 10580 consensus_peers.cc:485] T 930071b72771448d98cbe659ee23c63a P a8c4a518929d4bb3a88bdfb1649493bd -> Peer a885359a82bd4515913f7b273591f851 (127.9.176.193:38373): Couldn't send request to peer a885359a82bd4515913f7b273591f851. Status: Network error: Client connection negotiation failed: client connection to 127.9.176.193:38373: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:53:01.261134 10313 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.176.193:38373: connect: Connection refused (error 111)
W20240502 01:53:01.262493 10313 consensus_peers.cc:485] T 968bb8eb10d44a29b33299898ebe453c P ede1dd598e3541138a19a3ea5c052bc9 -> Peer a885359a82bd4515913f7b273591f851 (127.9.176.193:38373): Couldn't send request to peer a885359a82bd4515913f7b273591f851. Status: Network error: Client connection negotiation failed: client connection to 127.9.176.193:38373: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:01.268325 10285 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 01:53:01.302310 10580 consensus_peers.cc:485] T 8fcb99ef99114ca59130ccef6d0803b3 P a8c4a518929d4bb3a88bdfb1649493bd -> Peer a885359a82bd4515913f7b273591f851 (127.9.176.193:38373): Couldn't send request to peer a885359a82bd4515913f7b273591f851. Status: Network error: Client connection negotiation failed: client connection to 127.9.176.193:38373: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:53:01.351109 10580 consensus_peers.cc:485] T c9f395cabd05425484aad0f82b0a3ca9 P a8c4a518929d4bb3a88bdfb1649493bd -> Peer a885359a82bd4515913f7b273591f851 (127.9.176.193:38373): Couldn't send request to peer a885359a82bd4515913f7b273591f851. Status: Network error: Client connection negotiation failed: client connection to 127.9.176.193:38373: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:53:01.357228 10714 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.176.193:38373: connect: Connection refused (error 111)
W20240502 01:53:01.358860 10714 consensus_peers.cc:485] T a97bdeed6d00481d8a352a67da51c7f6 P fd95bcd4cdcb4cc5966050ffe2cffdff -> Peer a885359a82bd4515913f7b273591f851 (127.9.176.193:38373): Couldn't send request to peer a885359a82bd4515913f7b273591f851. Status: Network error: Client connection negotiation failed: client connection to 127.9.176.193:38373: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:53:01.393049 10580 consensus_peers.cc:485] T 941430c4941d4ce7bf4252f0b56abecc P a8c4a518929d4bb3a88bdfb1649493bd -> Peer a885359a82bd4515913f7b273591f851 (127.9.176.193:38373): Couldn't send request to peer a885359a82bd4515913f7b273591f851. Status: Network error: Client connection negotiation failed: client connection to 127.9.176.193:38373: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:01.410920  9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu with pid 10161
W20240502 01:53:01.435693 10712 connection.cc:535] server connection from 127.9.176.194:56847 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:53:01.436185 10711 connection.cc:535] client connection to 127.9.176.194:43645 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 01:53:01.436380  9923 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu pid 10294
W20240502 01:53:01.437521 10711 consensus_peers.cc:485] T 3a4f610cdc9d4dd48fd44924e406d929 P fd95bcd4cdcb4cc5966050ffe2cffdff -> Peer 7b7a47245f294ce9a264fcc8f6351c0f (127.9.176.194:43645): Couldn't send request to peer 7b7a47245f294ce9a264fcc8f6351c0f. Status: Network error: Client connection negotiation failed: client connection to 127.9.176.194:43645: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:53:01.437728 10711 consensus_peers.cc:485] T f72ee833c1d54106802dd9191e142284 P fd95bcd4cdcb4cc5966050ffe2cffdff -> Peer 7b7a47245f294ce9a264fcc8f6351c0f (127.9.176.194:43645): Couldn't send request to peer 7b7a47245f294ce9a264fcc8f6351c0f. Status: Network error: Client connection negotiation failed: client connection to 127.9.176.194:43645: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:53:01.437814 10711 consensus_peers.cc:485] T ec9d18ab22e1466c8daa6b554ccaeebb P fd95bcd4cdcb4cc5966050ffe2cffdff -> Peer 7b7a47245f294ce9a264fcc8f6351c0f (127.9.176.194:43645): Couldn't send request to peer 7b7a47245f294ce9a264fcc8f6351c0f. Status: Network error: Client connection negotiation failed: client connection to 127.9.176.194:43645: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:53:01.437880 10711 consensus_peers.cc:485] T 09a892cf118d41d4bfffc1576731d54c P fd95bcd4cdcb4cc5966050ffe2cffdff -> Peer 7b7a47245f294ce9a264fcc8f6351c0f (127.9.176.194:43645): Couldn't send request to peer 7b7a47245f294ce9a264fcc8f6351c0f. Status: Network error: Client connection negotiation failed: client connection to 127.9.176.194:43645: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:53:01.441543 10443 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.176.194:43645: connect: Connection refused (error 111)
W20240502 01:53:01.443389 10443 consensus_peers.cc:485] T e71b5c9725d9400d8ed9faad7276a474 P 34da849acb2045a9a2ef03933592dd08 -> Peer 7b7a47245f294ce9a264fcc8f6351c0f (127.9.176.194:43645): Couldn't send request to peer 7b7a47245f294ce9a264fcc8f6351c0f. Status: Network error: Client connection negotiation failed: client connection to 127.9.176.194:43645: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:01.443850 10419 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 01:53:01.447068 10443 consensus_peers.cc:485] T 4c79cbf474df4ff0974e5611fbba66cd P 34da849acb2045a9a2ef03933592dd08 -> Peer 7b7a47245f294ce9a264fcc8f6351c0f (127.9.176.194:43645): Couldn't send request to peer 7b7a47245f294ce9a264fcc8f6351c0f. Status: Network error: Client connection negotiation failed: client connection to 127.9.176.194:43645: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:53:01.542729 10711 consensus_peers.cc:485] T 1da6b311a749427f9fdd394bec7a033f P fd95bcd4cdcb4cc5966050ffe2cffdff -> Peer 7b7a47245f294ce9a264fcc8f6351c0f (127.9.176.194:43645): Couldn't send request to peer 7b7a47245f294ce9a264fcc8f6351c0f. Status: Network error: Client connection negotiation failed: client connection to 127.9.176.194:43645: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:53:01.556185 10443 consensus_peers.cc:485] T ad11d50337ab43fdbc110dc200035fd1 P 34da849acb2045a9a2ef03933592dd08 -> Peer 7b7a47245f294ce9a264fcc8f6351c0f (127.9.176.194:43645): Couldn't send request to peer 7b7a47245f294ce9a264fcc8f6351c0f. Status: Network error: Client connection negotiation failed: client connection to 127.9.176.194:43645: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:53:01.556859 10443 consensus_peers.cc:485] T c2730c626fe34084ab71d463dd2b8656 P 34da849acb2045a9a2ef03933592dd08 -> Peer 7b7a47245f294ce9a264fcc8f6351c0f (127.9.176.194:43645): Couldn't send request to peer 7b7a47245f294ce9a264fcc8f6351c0f. Status: Network error: Client connection negotiation failed: client connection to 127.9.176.194:43645: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:53:01.566507 10711 consensus_peers.cc:485] T a92d5a681b364e50896bdda0d92bb10a P fd95bcd4cdcb4cc5966050ffe2cffdff -> Peer 7b7a47245f294ce9a264fcc8f6351c0f (127.9.176.194:43645): Couldn't send request to peer 7b7a47245f294ce9a264fcc8f6351c0f. Status: Network error: Client connection negotiation failed: client connection to 127.9.176.194:43645: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:53:01.572971 10443 consensus_peers.cc:485] T c264070710e8405ab9c2b3fe2cea46ae P 34da849acb2045a9a2ef03933592dd08 -> Peer 7b7a47245f294ce9a264fcc8f6351c0f (127.9.176.194:43645): Couldn't send request to peer 7b7a47245f294ce9a264fcc8f6351c0f. Status: Network error: Client connection negotiation failed: client connection to 127.9.176.194:43645: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:01.585696  9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu with pid 10294
W20240502 01:53:01.610013 10579 connection.cc:535] client connection to 127.9.176.195:38701 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 01:53:01.610420  9923 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu pid 10427
W20240502 01:53:01.611428 10579 consensus_peers.cc:485] T b7218b5b2e2c4e1a862705edd8574f6b P a8c4a518929d4bb3a88bdfb1649493bd -> Peer ede1dd598e3541138a19a3ea5c052bc9 (127.9.176.195:38701): Couldn't send request to peer ede1dd598e3541138a19a3ea5c052bc9. Status: Network error: Client connection negotiation failed: client connection to 127.9.176.195:38701: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:01.616771 10552 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 01:53:01.663936 10577 consensus_peers.cc:485] T 75d03e64bee04a2092364ca91241cf4f P a8c4a518929d4bb3a88bdfb1649493bd -> Peer 7b7a47245f294ce9a264fcc8f6351c0f (127.9.176.194:43645): Couldn't send request to peer 7b7a47245f294ce9a264fcc8f6351c0f. Status: Network error: Client connection negotiation failed: client connection to 127.9.176.194:43645: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:53:01.702499 10579 consensus_peers.cc:485] T 8fcb99ef99114ca59130ccef6d0803b3 P a8c4a518929d4bb3a88bdfb1649493bd -> Peer ede1dd598e3541138a19a3ea5c052bc9 (127.9.176.195:38701): Couldn't send request to peer ede1dd598e3541138a19a3ea5c052bc9. Status: Network error: Client connection negotiation failed: client connection to 127.9.176.195:38701: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:53:01.718322 10711 consensus_peers.cc:485] T 816e798bf21641c1872dc88d2fd3e820 P fd95bcd4cdcb4cc5966050ffe2cffdff -> Peer 7b7a47245f294ce9a264fcc8f6351c0f (127.9.176.194:43645): Couldn't send request to peer 7b7a47245f294ce9a264fcc8f6351c0f. Status: Network error: Client connection negotiation failed: client connection to 127.9.176.194:43645: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:53:01.726634 10579 consensus_peers.cc:485] T 930071b72771448d98cbe659ee23c63a P a8c4a518929d4bb3a88bdfb1649493bd -> Peer ede1dd598e3541138a19a3ea5c052bc9 (127.9.176.195:38701): Couldn't send request to peer ede1dd598e3541138a19a3ea5c052bc9. Status: Network error: Client connection negotiation failed: client connection to 127.9.176.195:38701: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:53:01.726897 10711 consensus_peers.cc:485] T eaefd6b8be0947da8d8cb87644f91c70 P fd95bcd4cdcb4cc5966050ffe2cffdff -> Peer 7b7a47245f294ce9a264fcc8f6351c0f (127.9.176.194:43645): Couldn't send request to peer 7b7a47245f294ce9a264fcc8f6351c0f. Status: Network error: Client connection negotiation failed: client connection to 127.9.176.194:43645: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:01.761641  9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu with pid 10427
W20240502 01:53:01.783877 10579 consensus_peers.cc:485] T 941430c4941d4ce7bf4252f0b56abecc P a8c4a518929d4bb3a88bdfb1649493bd -> Peer ede1dd598e3541138a19a3ea5c052bc9 (127.9.176.195:38701): Couldn't send request to peer ede1dd598e3541138a19a3ea5c052bc9. Status: Network error: Client connection negotiation failed: client connection to 127.9.176.195:38701: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:53:01.788439 10713 connection.cc:535] server connection from 127.9.176.196:44873 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:53:01.789474 10711 consensus_peers.cc:485] T 3a4f610cdc9d4dd48fd44924e406d929 P fd95bcd4cdcb4cc5966050ffe2cffdff -> Peer 34da849acb2045a9a2ef03933592dd08 (127.9.176.196:44521): Couldn't send request to peer 34da849acb2045a9a2ef03933592dd08. Status: Network error: Client connection negotiation failed: client connection to 127.9.176.196:44521: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:53:01.789628 10711 consensus_peers.cc:485] T f72ee833c1d54106802dd9191e142284 P fd95bcd4cdcb4cc5966050ffe2cffdff -> Peer 34da849acb2045a9a2ef03933592dd08 (127.9.176.196:44521): Couldn't send request to peer 34da849acb2045a9a2ef03933592dd08. Status: Network error: Client connection negotiation failed: client connection to 127.9.176.196:44521: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:01.790138  9923 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu pid 10561
W20240502 01:53:01.791364 10579 consensus_peers.cc:485] T c9f395cabd05425484aad0f82b0a3ca9 P a8c4a518929d4bb3a88bdfb1649493bd -> Peer ede1dd598e3541138a19a3ea5c052bc9 (127.9.176.195:38701): Couldn't send request to peer ede1dd598e3541138a19a3ea5c052bc9. Status: Network error: Client connection negotiation failed: client connection to 127.9.176.195:38701: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:01.796792 10684 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 01:53:01.888574 10711 consensus_peers.cc:485] T a92d5a681b364e50896bdda0d92bb10a P fd95bcd4cdcb4cc5966050ffe2cffdff -> Peer 34da849acb2045a9a2ef03933592dd08 (127.9.176.196:44521): Couldn't send request to peer 34da849acb2045a9a2ef03933592dd08. Status: Network error: Client connection negotiation failed: client connection to 127.9.176.196:44521: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:53:01.908609 10713 consensus_peers.cc:485] T a97bdeed6d00481d8a352a67da51c7f6 P fd95bcd4cdcb4cc5966050ffe2cffdff -> Peer ede1dd598e3541138a19a3ea5c052bc9 (127.9.176.195:38701): Couldn't send request to peer ede1dd598e3541138a19a3ea5c052bc9. Status: Network error: Client connection negotiation failed: client connection to 127.9.176.195:38701: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:53:01.911612 10711 consensus_peers.cc:485] T 1da6b311a749427f9fdd394bec7a033f P fd95bcd4cdcb4cc5966050ffe2cffdff -> Peer 34da849acb2045a9a2ef03933592dd08 (127.9.176.196:44521): Couldn't send request to peer 34da849acb2045a9a2ef03933592dd08. Status: Network error: Client connection negotiation failed: client connection to 127.9.176.196:44521: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:01.942790  9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu with pid 10561
W20240502 01:53:01.970207  9954 connection.cc:535] server connection from 127.9.176.197:35521 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 01:53:01.970306  9923 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu pid 10694
W20240502 01:53:01.973840 10711 consensus_peers.cc:485] T 09a892cf118d41d4bfffc1576731d54c P fd95bcd4cdcb4cc5966050ffe2cffdff -> Peer 34da849acb2045a9a2ef03933592dd08 (127.9.176.196:44521): Couldn't send request to peer 34da849acb2045a9a2ef03933592dd08. Status: Network error: Client connection negotiation failed: client connection to 127.9.176.196:44521: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:01.977896 10821 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 01:53:02.120574 10711 consensus_peers.cc:485] T eaefd6b8be0947da8d8cb87644f91c70 P fd95bcd4cdcb4cc5966050ffe2cffdff -> Peer 34da849acb2045a9a2ef03933592dd08 (127.9.176.196:44521): Couldn't send request to peer 34da849acb2045a9a2ef03933592dd08. Status: Network error: Client connection negotiation failed: client connection to 127.9.176.196:44521: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:02.120709  9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu with pid 10694
W20240502 01:53:02.120752 10711 consensus_peers.cc:485] T 816e798bf21641c1872dc88d2fd3e820 P fd95bcd4cdcb4cc5966050ffe2cffdff -> Peer 34da849acb2045a9a2ef03933592dd08 (127.9.176.196:44521): Couldn't send request to peer 34da849acb2045a9a2ef03933592dd08. Status: Network error: Client connection negotiation failed: client connection to 127.9.176.196:44521: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:53:02.147048  9952 connection.cc:535] server connection from 127.9.176.198:49223 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 01:53:02.147403  9923 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu pid 9936
I20240502 01:53:02.148520  9998 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:53:02.309353  9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu with pid 9936
2024-05-02T01:53:02Z chronyd exiting
[       OK ] LocationAwareRebalancingBasicTest.Basic (31420 ms)
[----------] 1 test from LocationAwareRebalancingBasicTest (31421 ms total)

[----------] 1 test from RebalanceStartCriteriaTest
[ RUN      ] RebalanceStartCriteriaTest.UnknownIgnoredTServer/1
I20240502 01:53:02.469693  9923 test_util.cc:268] Using random seed: -586602997
I20240502 01:53:02.471963  9923 ts_itest-base.cc:112] Starting cluster with:
I20240502 01:53:02.472093  9923 ts_itest-base.cc:113] --------------
I20240502 01:53:02.472151  9923 ts_itest-base.cc:114] 5 tablet servers
I20240502 01:53:02.472239  9923 ts_itest-base.cc:115] 3 replicas per TS
I20240502 01:53:02.472286  9923 ts_itest-base.cc:116] --------------
2024-05-02T01:53:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T01:53:02Z Disabled control of system clock
I20240502 01:53:02.512001  9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
/tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:37861
--webserver_interface=127.9.176.254
--webserver_port=0
--builtin_ntp_servers=127.9.176.212:46583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.176.254:37861
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240502 01:53:02.858559 12311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:02.858908 12311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:53:02.859001 12311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:02.866801 12311 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:53:02.866967 12311 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 01:53:02.867020 12311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:53:02.867082 12311 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 01:53:02.867131 12311 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 01:53:02.876746 12311 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:46583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.9.176.254:37861
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:37861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.9.176.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true

Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:53:02.878134 12311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:53:02.880043 12311 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:53:02.892017 12317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:02.892650 12316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:02.893255 12319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:53:02.893929 12311 server_base.cc:1034] running on GCE node
I20240502 01:53:02.894783 12311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:53:02.896445 12311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:53:02.897647 12311 hybrid_clock.cc:648] HybridClock initialized: now 1714614782897611 us; error 40 us; skew 500 ppm
I20240502 01:53:02.897981 12311 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:53:02.900274 12311 webserver.cc:460] Webserver started at http://127.9.176.254:34483/ using document root <none> and password file <none>
I20240502 01:53:02.901021 12311 fs_manager.cc:359] Metadata directory not provided
I20240502 01:53:02.901140 12311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:53:02.901476 12311 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:53:02.904146 12311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "305b6d2414b84f6bab0c7f597355ad66"
format_stamp: "Formatted at 2024-05-02 01:53:02 on dist-test-slave-j3hh"
I20240502 01:53:02.904897 12311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "305b6d2414b84f6bab0c7f597355ad66"
format_stamp: "Formatted at 2024-05-02 01:53:02 on dist-test-slave-j3hh"
I20240502 01:53:02.911345 12311 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.006s	sys 0.000s
I20240502 01:53:02.914906 12325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:53:02.916297 12311 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240502 01:53:02.916474 12311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/wal
uuid: "305b6d2414b84f6bab0c7f597355ad66"
format_stamp: "Formatted at 2024-05-02 01:53:02 on dist-test-slave-j3hh"
I20240502 01:53:02.916673 12311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:53:02.945068 12311 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:53:02.945986 12311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:53:02.946272 12311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:53:02.965446 12311 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.254:37861
I20240502 01:53:02.965467 12377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.254:37861 every 8 connection(s)
I20240502 01:53:02.966930 12311 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 01:53:02.967497  9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu as pid 12311
I20240502 01:53:02.967756  9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 01:53:02.971796 12378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:02.982805 12378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 305b6d2414b84f6bab0c7f597355ad66: Bootstrap starting.
I20240502 01:53:02.986198 12378 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 305b6d2414b84f6bab0c7f597355ad66: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:02.987351 12378 log.cc:824] T 00000000000000000000000000000000 P 305b6d2414b84f6bab0c7f597355ad66: Log is configured to *not* fsync() on all Append() calls
I20240502 01:53:02.990389 12378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 305b6d2414b84f6bab0c7f597355ad66: No bootstrap required, opened a new log
I20240502 01:53:02.997081 12378 raft_consensus.cc:348] T 00000000000000000000000000000000 P 305b6d2414b84f6bab0c7f597355ad66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "305b6d2414b84f6bab0c7f597355ad66" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 37861 } }
I20240502 01:53:02.997493 12378 raft_consensus.cc:374] T 00000000000000000000000000000000 P 305b6d2414b84f6bab0c7f597355ad66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:02.997601 12378 raft_consensus.cc:729] T 00000000000000000000000000000000 P 305b6d2414b84f6bab0c7f597355ad66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 305b6d2414b84f6bab0c7f597355ad66, State: Initialized, Role: FOLLOWER
I20240502 01:53:02.998306 12378 consensus_queue.cc:260] T 00000000000000000000000000000000 P 305b6d2414b84f6bab0c7f597355ad66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "305b6d2414b84f6bab0c7f597355ad66" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 37861 } }
I20240502 01:53:02.998520 12378 raft_consensus.cc:388] T 00000000000000000000000000000000 P 305b6d2414b84f6bab0c7f597355ad66 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:53:02.998654 12378 raft_consensus.cc:482] T 00000000000000000000000000000000 P 305b6d2414b84f6bab0c7f597355ad66 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:53:02.998808 12378 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 305b6d2414b84f6bab0c7f597355ad66 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:03.001142 12378 raft_consensus.cc:504] T 00000000000000000000000000000000 P 305b6d2414b84f6bab0c7f597355ad66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "305b6d2414b84f6bab0c7f597355ad66" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 37861 } }
I20240502 01:53:03.001736 12378 leader_election.cc:302] T 00000000000000000000000000000000 P 305b6d2414b84f6bab0c7f597355ad66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 305b6d2414b84f6bab0c7f597355ad66; no voters: 
I20240502 01:53:03.002348 12378 leader_election.cc:288] T 00000000000000000000000000000000 P 305b6d2414b84f6bab0c7f597355ad66 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240502 01:53:03.002518 12383 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 305b6d2414b84f6bab0c7f597355ad66 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:03.003238 12383 raft_consensus.cc:686] T 00000000000000000000000000000000 P 305b6d2414b84f6bab0c7f597355ad66 [term 1 LEADER]: Becoming Leader. State: Replica: 305b6d2414b84f6bab0c7f597355ad66, State: Running, Role: LEADER
I20240502 01:53:03.003998 12383 consensus_queue.cc:237] T 00000000000000000000000000000000 P 305b6d2414b84f6bab0c7f597355ad66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "305b6d2414b84f6bab0c7f597355ad66" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 37861 } }
I20240502 01:53:03.004727 12378 sys_catalog.cc:564] T 00000000000000000000000000000000 P 305b6d2414b84f6bab0c7f597355ad66 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 01:53:03.007325 12384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 305b6d2414b84f6bab0c7f597355ad66 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "305b6d2414b84f6bab0c7f597355ad66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "305b6d2414b84f6bab0c7f597355ad66" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 37861 } } }
I20240502 01:53:03.007323 12385 sys_catalog.cc:455] T 00000000000000000000000000000000 P 305b6d2414b84f6bab0c7f597355ad66 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 305b6d2414b84f6bab0c7f597355ad66. Latest consensus state: current_term: 1 leader_uuid: "305b6d2414b84f6bab0c7f597355ad66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "305b6d2414b84f6bab0c7f597355ad66" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 37861 } } }
I20240502 01:53:03.007794 12384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 305b6d2414b84f6bab0c7f597355ad66 [sys.catalog]: This master's current role is: LEADER
I20240502 01:53:03.007828 12385 sys_catalog.cc:458] T 00000000000000000000000000000000 P 305b6d2414b84f6bab0c7f597355ad66 [sys.catalog]: This master's current role is: LEADER
I20240502 01:53:03.009124 12390 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 01:53:03.015388 12390 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 01:53:03.024056 12390 catalog_manager.cc:1347] Generated new cluster ID: 5bb825618676452d8405ce175a082424
I20240502 01:53:03.024189 12390 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 01:53:03.065078 12390 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 01:53:03.066922 12390 catalog_manager.cc:1504] Loading token signing keys...
I20240502 01:53:03.082487 12390 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 305b6d2414b84f6bab0c7f597355ad66: Generated new TSK 0
I20240502 01:53:03.083462 12390 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 01:53:03.096757  9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
/tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.193:0
--local_ip_for_outbound_sockets=127.9.176.193
--webserver_interface=127.9.176.193
--webserver_port=0
--tserver_master_addrs=127.9.176.254:37861
--builtin_ntp_servers=127.9.176.212:46583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 01:53:03.457368 12402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:03.457698 12402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:53:03.457842 12402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:03.466351 12402 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:53:03.466516 12402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:53:03.466701 12402 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.193
I20240502 01:53:03.477185 12402 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:46583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.9.176.193
--webserver_port=0
--tserver_master_addrs=127.9.176.254:37861
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.193
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:53:03.478636 12402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:53:03.480623 12402 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:53:03.493822 12407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:03.494401 12410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:03.494987 12408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:53:03.495604 12402 server_base.cc:1034] running on GCE node
I20240502 01:53:03.496308 12402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:53:03.497413 12402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:53:03.498659 12402 hybrid_clock.cc:648] HybridClock initialized: now 1714614783498596 us; error 62 us; skew 500 ppm
I20240502 01:53:03.499011 12402 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:53:03.501581 12402 webserver.cc:460] Webserver started at http://127.9.176.193:37373/ using document root <none> and password file <none>
I20240502 01:53:03.502395 12402 fs_manager.cc:359] Metadata directory not provided
I20240502 01:53:03.502526 12402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:53:03.502863 12402 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:53:03.505870 12402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "c57eb1856edb44bf8018097ec9fba04f"
format_stamp: "Formatted at 2024-05-02 01:53:03 on dist-test-slave-j3hh"
I20240502 01:53:03.506757 12402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "c57eb1856edb44bf8018097ec9fba04f"
format_stamp: "Formatted at 2024-05-02 01:53:03 on dist-test-slave-j3hh"
I20240502 01:53:03.512725 12402 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.002s	sys 0.005s
I20240502 01:53:03.516588 12416 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:53:03.517943 12402 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20240502 01:53:03.518122 12402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "c57eb1856edb44bf8018097ec9fba04f"
format_stamp: "Formatted at 2024-05-02 01:53:03 on dist-test-slave-j3hh"
I20240502 01:53:03.518337 12402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:53:03.538359 12402 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:53:03.539319 12402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:53:03.539681 12402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:53:03.540771 12402 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:53:03.542482 12402 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:53:03.542593 12402 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:53:03.542711 12402 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:53:03.542784 12402 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:53:03.576287 12402 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.193:35695
I20240502 01:53:03.576351 12529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.193:35695 every 8 connection(s)
I20240502 01:53:03.577705 12402 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 01:53:03.584935  9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu as pid 12402
I20240502 01:53:03.585187  9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 01:53:03.588861  9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
/tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.194:0
--local_ip_for_outbound_sockets=127.9.176.194
--webserver_interface=127.9.176.194
--webserver_port=0
--tserver_master_addrs=127.9.176.254:37861
--builtin_ntp_servers=127.9.176.212:46583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:53:03.592162 12530 heartbeater.cc:347] Connected to a master server at 127.9.176.254:37861
I20240502 01:53:03.592517 12530 heartbeater.cc:464] Registering TS with master...
I20240502 01:53:03.593410 12530 heartbeater.cc:511] Master 127.9.176.254:37861 requested a full tablet report, sending...
I20240502 01:53:03.595650 12342 ts_manager.cc:194] Registered new tserver with Master: c57eb1856edb44bf8018097ec9fba04f (127.9.176.193:35695)
I20240502 01:53:03.598053 12342 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.176.193:46319
W20240502 01:53:03.938859 12534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:03.939237 12534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:53:03.939330 12534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:03.947386 12534 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:53:03.947540 12534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:53:03.947701 12534 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.194
I20240502 01:53:03.957270 12534 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:46583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.9.176.194
--webserver_port=0
--tserver_master_addrs=127.9.176.254:37861
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.194
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:53:03.958976 12534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:53:03.961074 12534 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:53:03.975612 12539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:03.976037 12540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:03.979077 12542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:53:03.981259 12534 server_base.cc:1034] running on GCE node
I20240502 01:53:03.981971 12534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:53:03.983044 12534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:53:03.984284 12534 hybrid_clock.cc:648] HybridClock initialized: now 1714614783984243 us; error 37 us; skew 500 ppm
I20240502 01:53:03.984692 12534 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:53:03.987215 12534 webserver.cc:460] Webserver started at http://127.9.176.194:36341/ using document root <none> and password file <none>
I20240502 01:53:03.988143 12534 fs_manager.cc:359] Metadata directory not provided
I20240502 01:53:03.988278 12534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:53:03.988626 12534 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:53:03.991626 12534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "734645c232d145a3907cfea827207de1"
format_stamp: "Formatted at 2024-05-02 01:53:03 on dist-test-slave-j3hh"
I20240502 01:53:03.992632 12534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "734645c232d145a3907cfea827207de1"
format_stamp: "Formatted at 2024-05-02 01:53:03 on dist-test-slave-j3hh"
I20240502 01:53:03.998916 12534 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.005s	sys 0.001s
I20240502 01:53:04.003206 12548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:53:04.004837 12534 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.000s	sys 0.003s
I20240502 01:53:04.005106 12534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "734645c232d145a3907cfea827207de1"
format_stamp: "Formatted at 2024-05-02 01:53:03 on dist-test-slave-j3hh"
I20240502 01:53:04.005329 12534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:53:04.023087 12534 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:53:04.024127 12534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:53:04.024395 12534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:53:04.025424 12534 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:53:04.027020 12534 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:53:04.027117 12534 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:53:04.027208 12534 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:53:04.027263 12534 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:53:04.062247 12534 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.194:36303
I20240502 01:53:04.062297 12661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.194:36303 every 8 connection(s)
I20240502 01:53:04.063743 12534 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 01:53:04.066498  9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu as pid 12534
I20240502 01:53:04.066715  9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 01:53:04.069844  9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
/tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.195:0
--local_ip_for_outbound_sockets=127.9.176.195
--webserver_interface=127.9.176.195
--webserver_port=0
--tserver_master_addrs=127.9.176.254:37861
--builtin_ntp_servers=127.9.176.212:46583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:53:04.078333 12662 heartbeater.cc:347] Connected to a master server at 127.9.176.254:37861
I20240502 01:53:04.078670 12662 heartbeater.cc:464] Registering TS with master...
I20240502 01:53:04.079659 12662 heartbeater.cc:511] Master 127.9.176.254:37861 requested a full tablet report, sending...
I20240502 01:53:04.081463 12342 ts_manager.cc:194] Registered new tserver with Master: 734645c232d145a3907cfea827207de1 (127.9.176.194:36303)
I20240502 01:53:04.082674 12342 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.176.194:35389
W20240502 01:53:04.418800 12665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:04.419122 12665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:53:04.419234 12665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:04.427141 12665 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:53:04.427276 12665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:53:04.427438 12665 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.195
I20240502 01:53:04.436817 12665 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:46583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.9.176.195
--webserver_port=0
--tserver_master_addrs=127.9.176.254:37861
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.195
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:53:04.438261 12665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:53:04.440263 12665 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:53:04.453701 12670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:04.454281 12671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:04.454870 12673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:53:04.455500 12665 server_base.cc:1034] running on GCE node
I20240502 01:53:04.456264 12665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:53:04.457336 12665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:53:04.458549 12665 hybrid_clock.cc:648] HybridClock initialized: now 1714614784458491 us; error 51 us; skew 500 ppm
I20240502 01:53:04.458907 12665 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:53:04.461578 12665 webserver.cc:460] Webserver started at http://127.9.176.195:42463/ using document root <none> and password file <none>
I20240502 01:53:04.462442 12665 fs_manager.cc:359] Metadata directory not provided
I20240502 01:53:04.462564 12665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:53:04.462924 12665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:53:04.465837 12665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "cceb38a0ded14a03aa78fb34b3367067"
format_stamp: "Formatted at 2024-05-02 01:53:04 on dist-test-slave-j3hh"
I20240502 01:53:04.466742 12665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "cceb38a0ded14a03aa78fb34b3367067"
format_stamp: "Formatted at 2024-05-02 01:53:04 on dist-test-slave-j3hh"
I20240502 01:53:04.472841 12665 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.008s	sys 0.000s
I20240502 01:53:04.476989 12679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:53:04.478689 12665 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.001s
I20240502 01:53:04.478901 12665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "cceb38a0ded14a03aa78fb34b3367067"
format_stamp: "Formatted at 2024-05-02 01:53:04 on dist-test-slave-j3hh"
I20240502 01:53:04.479149 12665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:53:04.514317 12665 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:53:04.515211 12665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:53:04.515493 12665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:53:04.516618 12665 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:53:04.518249 12665 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:53:04.518338 12665 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:53:04.518426 12665 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:53:04.518482 12665 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:53:04.552704 12665 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.195:41715
I20240502 01:53:04.552795 12791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.195:41715 every 8 connection(s)
I20240502 01:53:04.554244 12665 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 01:53:04.558063  9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu as pid 12665
I20240502 01:53:04.558236  9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 01:53:04.561743  9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
/tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.196:0
--local_ip_for_outbound_sockets=127.9.176.196
--webserver_interface=127.9.176.196
--webserver_port=0
--tserver_master_addrs=127.9.176.254:37861
--builtin_ntp_servers=127.9.176.212:46583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:53:04.569726 12792 heartbeater.cc:347] Connected to a master server at 127.9.176.254:37861
I20240502 01:53:04.570114 12792 heartbeater.cc:464] Registering TS with master...
I20240502 01:53:04.571131 12792 heartbeater.cc:511] Master 127.9.176.254:37861 requested a full tablet report, sending...
I20240502 01:53:04.572916 12342 ts_manager.cc:194] Registered new tserver with Master: cceb38a0ded14a03aa78fb34b3367067 (127.9.176.195:41715)
I20240502 01:53:04.573828 12342 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.176.195:49281
I20240502 01:53:04.602028 12530 heartbeater.cc:503] Master 127.9.176.254:37861 was elected leader, sending a full tablet report...
W20240502 01:53:04.930038 12796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:04.930389 12796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:53:04.930521 12796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:04.938573 12796 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:53:04.938748 12796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:53:04.938908 12796 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.196
I20240502 01:53:04.948669 12796 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:46583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.9.176.196
--webserver_port=0
--tserver_master_addrs=127.9.176.254:37861
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.196
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:53:04.950114 12796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:53:04.952047 12796 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:53:04.965673 12801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:04.966192 12802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:04.966787 12804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:53:04.967391 12796 server_base.cc:1034] running on GCE node
I20240502 01:53:04.968147 12796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:53:04.969262 12796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:53:04.970513 12796 hybrid_clock.cc:648] HybridClock initialized: now 1714614784970439 us; error 68 us; skew 500 ppm
I20240502 01:53:04.970990 12796 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:53:04.973948 12796 webserver.cc:460] Webserver started at http://127.9.176.196:36997/ using document root <none> and password file <none>
I20240502 01:53:04.974839 12796 fs_manager.cc:359] Metadata directory not provided
I20240502 01:53:04.975000 12796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:53:04.975349 12796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:53:04.978371 12796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "4c73875a8014497b8c329b618d44dcd0"
format_stamp: "Formatted at 2024-05-02 01:53:04 on dist-test-slave-j3hh"
I20240502 01:53:04.979297 12796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "4c73875a8014497b8c329b618d44dcd0"
format_stamp: "Formatted at 2024-05-02 01:53:04 on dist-test-slave-j3hh"
I20240502 01:53:04.985641 12796 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.004s	sys 0.000s
I20240502 01:53:04.989923 12810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:53:04.991680 12796 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.000s
I20240502 01:53:04.991880 12796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "4c73875a8014497b8c329b618d44dcd0"
format_stamp: "Formatted at 2024-05-02 01:53:04 on dist-test-slave-j3hh"
I20240502 01:53:04.992118 12796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:53:05.009529 12796 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:53:05.010628 12796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:53:05.010943 12796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:53:05.012048 12796 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:53:05.013754 12796 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:53:05.013850 12796 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:53:05.013943 12796 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:53:05.014000 12796 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:53:05.049823 12796 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.196:43141
I20240502 01:53:05.049867 12922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.196:43141 every 8 connection(s)
I20240502 01:53:05.051319 12796 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 01:53:05.059844  9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu as pid 12796
I20240502 01:53:05.060053  9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 01:53:05.064079  9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
/tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.197:0
--local_ip_for_outbound_sockets=127.9.176.197
--webserver_interface=127.9.176.197
--webserver_port=0
--tserver_master_addrs=127.9.176.254:37861
--builtin_ntp_servers=127.9.176.212:46583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:53:05.066555 12923 heartbeater.cc:347] Connected to a master server at 127.9.176.254:37861
I20240502 01:53:05.066960 12923 heartbeater.cc:464] Registering TS with master...
I20240502 01:53:05.068228 12923 heartbeater.cc:511] Master 127.9.176.254:37861 requested a full tablet report, sending...
I20240502 01:53:05.070192 12342 ts_manager.cc:194] Registered new tserver with Master: 4c73875a8014497b8c329b618d44dcd0 (127.9.176.196:43141)
I20240502 01:53:05.071261 12342 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.176.196:47159
I20240502 01:53:05.085633 12662 heartbeater.cc:503] Master 127.9.176.254:37861 was elected leader, sending a full tablet report...
W20240502 01:53:05.419991 12926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:05.420293 12926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:53:05.420429 12926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:05.428282 12926 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:53:05.428436 12926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:53:05.428599 12926 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.197
I20240502 01:53:05.438016 12926 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:46583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.197:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.9.176.197
--webserver_port=0
--tserver_master_addrs=127.9.176.254:37861
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.197
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:53:05.439424 12926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:53:05.441327 12926 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:53:05.453943 12931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:05.454579 12932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:05.455107 12934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:53:05.455904 12926 server_base.cc:1034] running on GCE node
I20240502 01:53:05.456642 12926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:53:05.457763 12926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:53:05.459024 12926 hybrid_clock.cc:648] HybridClock initialized: now 1714614785458966 us; error 47 us; skew 500 ppm
I20240502 01:53:05.459415 12926 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:53:05.462121 12926 webserver.cc:460] Webserver started at http://127.9.176.197:39039/ using document root <none> and password file <none>
I20240502 01:53:05.462949 12926 fs_manager.cc:359] Metadata directory not provided
I20240502 01:53:05.463097 12926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:53:05.463459 12926 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:53:05.466425 12926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "debbe621f1a74c7e97c5cf37ccb6457f"
format_stamp: "Formatted at 2024-05-02 01:53:05 on dist-test-slave-j3hh"
I20240502 01:53:05.467355 12926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "debbe621f1a74c7e97c5cf37ccb6457f"
format_stamp: "Formatted at 2024-05-02 01:53:05 on dist-test-slave-j3hh"
I20240502 01:53:05.473628 12926 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.004s	sys 0.001s
I20240502 01:53:05.477550 12940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:53:05.479048 12926 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20240502 01:53:05.479244 12926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "debbe621f1a74c7e97c5cf37ccb6457f"
format_stamp: "Formatted at 2024-05-02 01:53:05 on dist-test-slave-j3hh"
I20240502 01:53:05.479473 12926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:53:05.499447 12926 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:53:05.500442 12926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:53:05.500730 12926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:53:05.501768 12926 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:53:05.503432 12926 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:53:05.503530 12926 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:53:05.503675 12926 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:53:05.503756 12926 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:53:05.539675 12926 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.197:42983
I20240502 01:53:05.539803 13054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.197:42983 every 8 connection(s)
I20240502 01:53:05.541237 12926 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 01:53:05.542331  9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu as pid 12926
I20240502 01:53:05.542515  9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 01:53:05.555012 13055 heartbeater.cc:347] Connected to a master server at 127.9.176.254:37861
I20240502 01:53:05.555416 13055 heartbeater.cc:464] Registering TS with master...
I20240502 01:53:05.556588 13055 heartbeater.cc:511] Master 127.9.176.254:37861 requested a full tablet report, sending...
I20240502 01:53:05.558338 12342 ts_manager.cc:194] Registered new tserver with Master: debbe621f1a74c7e97c5cf37ccb6457f (127.9.176.197:42983)
I20240502 01:53:05.559370  9923 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240502 01:53:05.559407 12342 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.176.197:37697
I20240502 01:53:05.577103 12792 heartbeater.cc:503] Master 127.9.176.254:37861 was elected leader, sending a full tablet report...
I20240502 01:53:05.577643 12342 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44132:
name: "TestTable"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
owner: "alice"
I20240502 01:53:05.621577 12596 tablet_service.cc:1467] Processing CreateTablet for tablet 252aa22ad82745f5b0da5e5ea381926a (DEFAULT_TABLE table=TestTable [id=246e9e362ca54ecbb2ea0e9559175501]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:05.621737 12464 tablet_service.cc:1467] Processing CreateTablet for tablet 252aa22ad82745f5b0da5e5ea381926a (DEFAULT_TABLE table=TestTable [id=246e9e362ca54ecbb2ea0e9559175501]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:05.623193 12596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 252aa22ad82745f5b0da5e5ea381926a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:05.623540 12464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 252aa22ad82745f5b0da5e5ea381926a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:05.632607 12989 tablet_service.cc:1467] Processing CreateTablet for tablet 252aa22ad82745f5b0da5e5ea381926a (DEFAULT_TABLE table=TestTable [id=246e9e362ca54ecbb2ea0e9559175501]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:05.634368 12989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 252aa22ad82745f5b0da5e5ea381926a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:05.635895 13074 tablet_bootstrap.cc:492] T 252aa22ad82745f5b0da5e5ea381926a P 734645c232d145a3907cfea827207de1: Bootstrap starting.
I20240502 01:53:05.638015 13075 tablet_bootstrap.cc:492] T 252aa22ad82745f5b0da5e5ea381926a P c57eb1856edb44bf8018097ec9fba04f: Bootstrap starting.
I20240502 01:53:05.640743 13074 tablet_bootstrap.cc:654] T 252aa22ad82745f5b0da5e5ea381926a P 734645c232d145a3907cfea827207de1: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:05.641897 13074 log.cc:824] T 252aa22ad82745f5b0da5e5ea381926a P 734645c232d145a3907cfea827207de1: Log is configured to *not* fsync() on all Append() calls
I20240502 01:53:05.641994 13075 tablet_bootstrap.cc:654] T 252aa22ad82745f5b0da5e5ea381926a P c57eb1856edb44bf8018097ec9fba04f: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:05.643441 13075 log.cc:824] T 252aa22ad82745f5b0da5e5ea381926a P c57eb1856edb44bf8018097ec9fba04f: Log is configured to *not* fsync() on all Append() calls
I20240502 01:53:05.644757 13074 tablet_bootstrap.cc:492] T 252aa22ad82745f5b0da5e5ea381926a P 734645c232d145a3907cfea827207de1: No bootstrap required, opened a new log
I20240502 01:53:05.645030 13074 ts_tablet_manager.cc:1367] T 252aa22ad82745f5b0da5e5ea381926a P 734645c232d145a3907cfea827207de1: Time spent bootstrapping tablet: real 0.010s	user 0.007s	sys 0.000s
I20240502 01:53:05.646476 13075 tablet_bootstrap.cc:492] T 252aa22ad82745f5b0da5e5ea381926a P c57eb1856edb44bf8018097ec9fba04f: No bootstrap required, opened a new log
I20240502 01:53:05.646788 13075 ts_tablet_manager.cc:1367] T 252aa22ad82745f5b0da5e5ea381926a P c57eb1856edb44bf8018097ec9fba04f: Time spent bootstrapping tablet: real 0.009s	user 0.003s	sys 0.003s
I20240502 01:53:05.649022 13078 tablet_bootstrap.cc:492] T 252aa22ad82745f5b0da5e5ea381926a P debbe621f1a74c7e97c5cf37ccb6457f: Bootstrap starting.
I20240502 01:53:05.653100 13074 raft_consensus.cc:348] T 252aa22ad82745f5b0da5e5ea381926a P 734645c232d145a3907cfea827207de1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "734645c232d145a3907cfea827207de1" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36303 } } peers { permanent_uuid: "debbe621f1a74c7e97c5cf37ccb6457f" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 42983 } } peers { permanent_uuid: "c57eb1856edb44bf8018097ec9fba04f" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35695 } }
I20240502 01:53:05.653666 13074 raft_consensus.cc:374] T 252aa22ad82745f5b0da5e5ea381926a P 734645c232d145a3907cfea827207de1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:05.653875 13074 raft_consensus.cc:729] T 252aa22ad82745f5b0da5e5ea381926a P 734645c232d145a3907cfea827207de1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 734645c232d145a3907cfea827207de1, State: Initialized, Role: FOLLOWER
I20240502 01:53:05.654718 13074 consensus_queue.cc:260] T 252aa22ad82745f5b0da5e5ea381926a P 734645c232d145a3907cfea827207de1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "734645c232d145a3907cfea827207de1" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36303 } } peers { permanent_uuid: "debbe621f1a74c7e97c5cf37ccb6457f" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 42983 } } peers { permanent_uuid: "c57eb1856edb44bf8018097ec9fba04f" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35695 } }
I20240502 01:53:05.654986 13078 tablet_bootstrap.cc:654] T 252aa22ad82745f5b0da5e5ea381926a P debbe621f1a74c7e97c5cf37ccb6457f: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:05.654835 13075 raft_consensus.cc:348] T 252aa22ad82745f5b0da5e5ea381926a P c57eb1856edb44bf8018097ec9fba04f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "734645c232d145a3907cfea827207de1" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36303 } } peers { permanent_uuid: "debbe621f1a74c7e97c5cf37ccb6457f" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 42983 } } peers { permanent_uuid: "c57eb1856edb44bf8018097ec9fba04f" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35695 } }
I20240502 01:53:05.655288 13075 raft_consensus.cc:374] T 252aa22ad82745f5b0da5e5ea381926a P c57eb1856edb44bf8018097ec9fba04f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:05.655483 13075 raft_consensus.cc:729] T 252aa22ad82745f5b0da5e5ea381926a P c57eb1856edb44bf8018097ec9fba04f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c57eb1856edb44bf8018097ec9fba04f, State: Initialized, Role: FOLLOWER
I20240502 01:53:05.656492 13075 consensus_queue.cc:260] T 252aa22ad82745f5b0da5e5ea381926a P c57eb1856edb44bf8018097ec9fba04f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "734645c232d145a3907cfea827207de1" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36303 } } peers { permanent_uuid: "debbe621f1a74c7e97c5cf37ccb6457f" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 42983 } } peers { permanent_uuid: "c57eb1856edb44bf8018097ec9fba04f" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35695 } }
I20240502 01:53:05.656630 13078 log.cc:824] T 252aa22ad82745f5b0da5e5ea381926a P debbe621f1a74c7e97c5cf37ccb6457f: Log is configured to *not* fsync() on all Append() calls
I20240502 01:53:05.656845 13074 ts_tablet_manager.cc:1398] T 252aa22ad82745f5b0da5e5ea381926a P 734645c232d145a3907cfea827207de1: Time spent starting tablet: real 0.012s	user 0.012s	sys 0.000s
I20240502 01:53:05.658977 13075 ts_tablet_manager.cc:1398] T 252aa22ad82745f5b0da5e5ea381926a P c57eb1856edb44bf8018097ec9fba04f: Time spent starting tablet: real 0.012s	user 0.004s	sys 0.008s
I20240502 01:53:05.660261 13078 tablet_bootstrap.cc:492] T 252aa22ad82745f5b0da5e5ea381926a P debbe621f1a74c7e97c5cf37ccb6457f: No bootstrap required, opened a new log
I20240502 01:53:05.660645 13078 ts_tablet_manager.cc:1367] T 252aa22ad82745f5b0da5e5ea381926a P debbe621f1a74c7e97c5cf37ccb6457f: Time spent bootstrapping tablet: real 0.012s	user 0.003s	sys 0.005s
I20240502 01:53:05.667955 13078 raft_consensus.cc:348] T 252aa22ad82745f5b0da5e5ea381926a P debbe621f1a74c7e97c5cf37ccb6457f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "734645c232d145a3907cfea827207de1" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36303 } } peers { permanent_uuid: "debbe621f1a74c7e97c5cf37ccb6457f" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 42983 } } peers { permanent_uuid: "c57eb1856edb44bf8018097ec9fba04f" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35695 } }
I20240502 01:53:05.668370 13078 raft_consensus.cc:374] T 252aa22ad82745f5b0da5e5ea381926a P debbe621f1a74c7e97c5cf37ccb6457f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:05.668460 13078 raft_consensus.cc:729] T 252aa22ad82745f5b0da5e5ea381926a P debbe621f1a74c7e97c5cf37ccb6457f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: debbe621f1a74c7e97c5cf37ccb6457f, State: Initialized, Role: FOLLOWER
I20240502 01:53:05.669101 13078 consensus_queue.cc:260] T 252aa22ad82745f5b0da5e5ea381926a P debbe621f1a74c7e97c5cf37ccb6457f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "734645c232d145a3907cfea827207de1" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36303 } } peers { permanent_uuid: "debbe621f1a74c7e97c5cf37ccb6457f" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 42983 } } peers { permanent_uuid: "c57eb1856edb44bf8018097ec9fba04f" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35695 } }
I20240502 01:53:05.670399 13055 heartbeater.cc:503] Master 127.9.176.254:37861 was elected leader, sending a full tablet report...
I20240502 01:53:05.671110 13078 ts_tablet_manager.cc:1398] T 252aa22ad82745f5b0da5e5ea381926a P debbe621f1a74c7e97c5cf37ccb6457f: Time spent starting tablet: real 0.010s	user 0.008s	sys 0.002s
I20240502 01:53:05.774225 13079 raft_consensus.cc:482] T 252aa22ad82745f5b0da5e5ea381926a P 734645c232d145a3907cfea827207de1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:05.774518 13079 raft_consensus.cc:504] T 252aa22ad82745f5b0da5e5ea381926a P 734645c232d145a3907cfea827207de1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "734645c232d145a3907cfea827207de1" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36303 } } peers { permanent_uuid: "debbe621f1a74c7e97c5cf37ccb6457f" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 42983 } } peers { permanent_uuid: "c57eb1856edb44bf8018097ec9fba04f" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35695 } }
I20240502 01:53:05.776394 13079 leader_election.cc:288] T 252aa22ad82745f5b0da5e5ea381926a P 734645c232d145a3907cfea827207de1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers debbe621f1a74c7e97c5cf37ccb6457f (127.9.176.197:42983), c57eb1856edb44bf8018097ec9fba04f (127.9.176.193:35695)
I20240502 01:53:05.783954 12485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "252aa22ad82745f5b0da5e5ea381926a" candidate_uuid: "734645c232d145a3907cfea827207de1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c57eb1856edb44bf8018097ec9fba04f" is_pre_election: true
I20240502 01:53:05.784686 12485 raft_consensus.cc:2454] T 252aa22ad82745f5b0da5e5ea381926a P c57eb1856edb44bf8018097ec9fba04f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 734645c232d145a3907cfea827207de1 in term 0.
I20240502 01:53:05.784932 13010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "252aa22ad82745f5b0da5e5ea381926a" candidate_uuid: "734645c232d145a3907cfea827207de1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "debbe621f1a74c7e97c5cf37ccb6457f" is_pre_election: true
I20240502 01:53:05.785538 13010 raft_consensus.cc:2454] T 252aa22ad82745f5b0da5e5ea381926a P debbe621f1a74c7e97c5cf37ccb6457f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 734645c232d145a3907cfea827207de1 in term 0.
I20240502 01:53:05.785588 12552 leader_election.cc:302] T 252aa22ad82745f5b0da5e5ea381926a P 734645c232d145a3907cfea827207de1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 734645c232d145a3907cfea827207de1, c57eb1856edb44bf8018097ec9fba04f; no voters: 
I20240502 01:53:05.786084 13079 raft_consensus.cc:2781] T 252aa22ad82745f5b0da5e5ea381926a P 734645c232d145a3907cfea827207de1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:05.786262 13079 raft_consensus.cc:482] T 252aa22ad82745f5b0da5e5ea381926a P 734645c232d145a3907cfea827207de1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:05.786370 13079 raft_consensus.cc:3037] T 252aa22ad82745f5b0da5e5ea381926a P 734645c232d145a3907cfea827207de1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:05.788589 13079 raft_consensus.cc:504] T 252aa22ad82745f5b0da5e5ea381926a P 734645c232d145a3907cfea827207de1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "734645c232d145a3907cfea827207de1" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36303 } } peers { permanent_uuid: "debbe621f1a74c7e97c5cf37ccb6457f" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 42983 } } peers { permanent_uuid: "c57eb1856edb44bf8018097ec9fba04f" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35695 } }
I20240502 01:53:05.789284 13079 leader_election.cc:288] T 252aa22ad82745f5b0da5e5ea381926a P 734645c232d145a3907cfea827207de1 [CANDIDATE]: Term 1 election: Requested vote from peers debbe621f1a74c7e97c5cf37ccb6457f (127.9.176.197:42983), c57eb1856edb44bf8018097ec9fba04f (127.9.176.193:35695)
I20240502 01:53:05.789839 12485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "252aa22ad82745f5b0da5e5ea381926a" candidate_uuid: "734645c232d145a3907cfea827207de1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c57eb1856edb44bf8018097ec9fba04f"
I20240502 01:53:05.789844 13010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "252aa22ad82745f5b0da5e5ea381926a" candidate_uuid: "734645c232d145a3907cfea827207de1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "debbe621f1a74c7e97c5cf37ccb6457f"
I20240502 01:53:05.790045 12485 raft_consensus.cc:3037] T 252aa22ad82745f5b0da5e5ea381926a P c57eb1856edb44bf8018097ec9fba04f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:05.790045 13010 raft_consensus.cc:3037] T 252aa22ad82745f5b0da5e5ea381926a P debbe621f1a74c7e97c5cf37ccb6457f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:05.792769 13010 raft_consensus.cc:2454] T 252aa22ad82745f5b0da5e5ea381926a P debbe621f1a74c7e97c5cf37ccb6457f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 734645c232d145a3907cfea827207de1 in term 1.
I20240502 01:53:05.792754 12485 raft_consensus.cc:2454] T 252aa22ad82745f5b0da5e5ea381926a P c57eb1856edb44bf8018097ec9fba04f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 734645c232d145a3907cfea827207de1 in term 1.
W20240502 01:53:05.793108 13056 tablet.cc:2369] T 252aa22ad82745f5b0da5e5ea381926a P debbe621f1a74c7e97c5cf37ccb6457f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:53:05.793270 12552 leader_election.cc:302] T 252aa22ad82745f5b0da5e5ea381926a P 734645c232d145a3907cfea827207de1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 734645c232d145a3907cfea827207de1, c57eb1856edb44bf8018097ec9fba04f; no voters: 
I20240502 01:53:05.793649 13079 raft_consensus.cc:2781] T 252aa22ad82745f5b0da5e5ea381926a P 734645c232d145a3907cfea827207de1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:05.794160 13079 raft_consensus.cc:686] T 252aa22ad82745f5b0da5e5ea381926a P 734645c232d145a3907cfea827207de1 [term 1 LEADER]: Becoming Leader. State: Replica: 734645c232d145a3907cfea827207de1, State: Running, Role: LEADER
I20240502 01:53:05.794710 13079 consensus_queue.cc:237] T 252aa22ad82745f5b0da5e5ea381926a P 734645c232d145a3907cfea827207de1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "734645c232d145a3907cfea827207de1" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36303 } } peers { permanent_uuid: "debbe621f1a74c7e97c5cf37ccb6457f" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 42983 } } peers { permanent_uuid: "c57eb1856edb44bf8018097ec9fba04f" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35695 } }
I20240502 01:53:05.799854 12341 catalog_manager.cc:5484] T 252aa22ad82745f5b0da5e5ea381926a P 734645c232d145a3907cfea827207de1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 734645c232d145a3907cfea827207de1 (127.9.176.194). New cstate: current_term: 1 leader_uuid: "734645c232d145a3907cfea827207de1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "734645c232d145a3907cfea827207de1" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36303 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "debbe621f1a74c7e97c5cf37ccb6457f" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 42983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c57eb1856edb44bf8018097ec9fba04f" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35695 } health_report { overall_health: UNKNOWN } } }
W20240502 01:53:05.816893 12663 tablet.cc:2369] T 252aa22ad82745f5b0da5e5ea381926a P 734645c232d145a3907cfea827207de1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 01:53:05.831298 12531 tablet.cc:2369] T 252aa22ad82745f5b0da5e5ea381926a P c57eb1856edb44bf8018097ec9fba04f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:53:05.844581  9923 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240502 01:53:05.847183  9923 ts_itest-base.cc:233] Waiting for 1 tablets on tserver c57eb1856edb44bf8018097ec9fba04f to finish bootstrapping
I20240502 01:53:05.856678  9923 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 734645c232d145a3907cfea827207de1 to finish bootstrapping
I20240502 01:53:05.865149  9923 ts_itest-base.cc:233] Waiting for 0 tablets on tserver cceb38a0ded14a03aa78fb34b3367067 to finish bootstrapping
I20240502 01:53:05.874727  9923 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 4c73875a8014497b8c329b618d44dcd0 to finish bootstrapping
I20240502 01:53:05.883636  9923 ts_itest-base.cc:233] Waiting for 1 tablets on tserver debbe621f1a74c7e97c5cf37ccb6457f to finish bootstrapping
I20240502 01:53:06.075273 12923 heartbeater.cc:503] Master 127.9.176.254:37861 was elected leader, sending a full tablet report...
I20240502 01:53:06.345887 13102 consensus_queue.cc:1035] T 252aa22ad82745f5b0da5e5ea381926a P 734645c232d145a3907cfea827207de1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "debbe621f1a74c7e97c5cf37ccb6457f" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 42983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:06.357843 13113 consensus_queue.cc:1035] T 252aa22ad82745f5b0da5e5ea381926a P 734645c232d145a3907cfea827207de1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57eb1856edb44bf8018097ec9fba04f" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:06.439824 12596 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:06.444737 12464 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:06.450163 12858 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:06.450698 12727 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:06.466609 12989 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:06.641311 12596 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:06.643826 12858 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:06.652093 12464 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:06.652175 12727 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:06.656424 12989 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:07.316313 12596 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:07.322914 12464 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:07.325223 12858 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:07.327896 12989 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:07.341671 12727 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:07.500905 12596 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:07.504529 12858 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:07.512259 12464 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:07.516564 12727 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:07.518482 12989 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:07.709857 12596 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:07.710083 12858 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:07.717455 12989 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:07.724424 12727 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:07.725672 12464 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:07.883080  9923 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu pid 12402
I20240502 01:53:07.884660 12525 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:53:07.983386  9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu with pid 12402
I20240502 01:53:08.002435  9923 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu pid 12534
I20240502 01:53:08.003495 12656 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:53:08.101239  9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu with pid 12534
I20240502 01:53:08.120592  9923 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu pid 12665
I20240502 01:53:08.121611 12786 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:53:08.213680  9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu with pid 12665
I20240502 01:53:08.233644  9923 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu pid 12796
I20240502 01:53:08.234735 12918 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:53:08.326521  9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu with pid 12796
I20240502 01:53:08.345325  9923 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu pid 12926
I20240502 01:53:08.346431 13050 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:53:08.443027  9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu with pid 12926
I20240502 01:53:08.463090  9923 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu pid 12311
W20240502 01:53:08.463048 12326 connection.cc:535] server connection from 127.9.176.197:37697 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 01:53:08.464247 12372 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:53:08.581187  9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu with pid 12311
2024-05-02T01:53:08Z chronyd exiting
[       OK ] RebalanceStartCriteriaTest.UnknownIgnoredTServer/1 (6155 ms)
[----------] 1 test from RebalanceStartCriteriaTest (6155 ms total)

[----------] 1 test from RebalanceParamTest
[ RUN      ] RebalanceParamTest.Rebalance/3
I20240502 01:53:08.625851  9923 test_util.cc:268] Using random seed: -580446814
I20240502 01:53:08.627846  9923 ts_itest-base.cc:112] Starting cluster with:
I20240502 01:53:08.627918  9923 ts_itest-base.cc:113] --------------
I20240502 01:53:08.627976  9923 ts_itest-base.cc:114] 7 tablet servers
I20240502 01:53:08.628018  9923 ts_itest-base.cc:115] 2 replicas per TS
I20240502 01:53:08.628068  9923 ts_itest-base.cc:116] --------------
2024-05-02T01:53:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T01:53:08Z Disabled control of system clock
I20240502 01:53:08.666954  9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
/tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:44447
--webserver_interface=127.9.176.254
--webserver_port=0
--builtin_ntp_servers=127.9.176.212:33337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.176.254:44447
--allow_unsafe_replication_factor
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--tserver_unresponsive_timeout_ms=3000 with env {}
W20240502 01:53:09.014259 13295 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240502 01:53:09.014622 13295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:09.014688 13295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:53:09.014763 13295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:09.022491 13295 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:53:09.022641 13295 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 01:53:09.022704 13295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:53:09.022744 13295 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 01:53:09.022810 13295 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 01:53:09.032438 13295 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:33337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.9.176.254:44447
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:44447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.9.176.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true

Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:53:09.033782 13295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:53:09.035596 13295 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:53:09.046795 13301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:09.047425 13300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:09.048060 13303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:53:09.048717 13295 server_base.cc:1034] running on GCE node
I20240502 01:53:09.049543 13295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:53:09.051057 13295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:53:09.052309 13295 hybrid_clock.cc:648] HybridClock initialized: now 1714614789052240 us; error 46 us; skew 500 ppm
I20240502 01:53:09.052681 13295 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:53:09.054983 13295 webserver.cc:460] Webserver started at http://127.9.176.254:34149/ using document root <none> and password file <none>
I20240502 01:53:09.055840 13295 fs_manager.cc:359] Metadata directory not provided
I20240502 01:53:09.055986 13295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:53:09.056313 13295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:53:09.059235 13295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "fad8aa713de7433db0f21791b371a3e4"
format_stamp: "Formatted at 2024-05-02 01:53:09 on dist-test-slave-j3hh"
I20240502 01:53:09.060094 13295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "fad8aa713de7433db0f21791b371a3e4"
format_stamp: "Formatted at 2024-05-02 01:53:09 on dist-test-slave-j3hh"
I20240502 01:53:09.066080 13295 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.002s	sys 0.004s
I20240502 01:53:09.069962 13309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:53:09.071452 13295 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20240502 01:53:09.071695 13295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/wal
uuid: "fad8aa713de7433db0f21791b371a3e4"
format_stamp: "Formatted at 2024-05-02 01:53:09 on dist-test-slave-j3hh"
I20240502 01:53:09.071903 13295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:53:09.100768 13295 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:53:09.101711 13295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:53:09.102020 13295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:53:09.121816 13295 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.254:44447
I20240502 01:53:09.121835 13360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.254:44447 every 8 connection(s)
I20240502 01:53:09.123414 13295 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 01:53:09.127377 13362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:09.133276  9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu as pid 13295
I20240502 01:53:09.133520  9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 01:53:09.138283 13362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fad8aa713de7433db0f21791b371a3e4: Bootstrap starting.
I20240502 01:53:09.141888 13362 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fad8aa713de7433db0f21791b371a3e4: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:09.143076 13362 log.cc:824] T 00000000000000000000000000000000 P fad8aa713de7433db0f21791b371a3e4: Log is configured to *not* fsync() on all Append() calls
I20240502 01:53:09.146139 13362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fad8aa713de7433db0f21791b371a3e4: No bootstrap required, opened a new log
I20240502 01:53:09.152387 13362 raft_consensus.cc:348] T 00000000000000000000000000000000 P fad8aa713de7433db0f21791b371a3e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fad8aa713de7433db0f21791b371a3e4" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 44447 } }
I20240502 01:53:09.152734 13362 raft_consensus.cc:374] T 00000000000000000000000000000000 P fad8aa713de7433db0f21791b371a3e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:09.152813 13362 raft_consensus.cc:729] T 00000000000000000000000000000000 P fad8aa713de7433db0f21791b371a3e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fad8aa713de7433db0f21791b371a3e4, State: Initialized, Role: FOLLOWER
I20240502 01:53:09.153357 13362 consensus_queue.cc:260] T 00000000000000000000000000000000 P fad8aa713de7433db0f21791b371a3e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fad8aa713de7433db0f21791b371a3e4" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 44447 } }
I20240502 01:53:09.153548 13362 raft_consensus.cc:388] T 00000000000000000000000000000000 P fad8aa713de7433db0f21791b371a3e4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:53:09.153646 13362 raft_consensus.cc:482] T 00000000000000000000000000000000 P fad8aa713de7433db0f21791b371a3e4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:53:09.153781 13362 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fad8aa713de7433db0f21791b371a3e4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:09.156040 13362 raft_consensus.cc:504] T 00000000000000000000000000000000 P fad8aa713de7433db0f21791b371a3e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fad8aa713de7433db0f21791b371a3e4" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 44447 } }
I20240502 01:53:09.156538 13362 leader_election.cc:302] T 00000000000000000000000000000000 P fad8aa713de7433db0f21791b371a3e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fad8aa713de7433db0f21791b371a3e4; no voters: 
I20240502 01:53:09.157122 13362 leader_election.cc:288] T 00000000000000000000000000000000 P fad8aa713de7433db0f21791b371a3e4 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240502 01:53:09.157251 13367 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fad8aa713de7433db0f21791b371a3e4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:09.157917 13367 raft_consensus.cc:686] T 00000000000000000000000000000000 P fad8aa713de7433db0f21791b371a3e4 [term 1 LEADER]: Becoming Leader. State: Replica: fad8aa713de7433db0f21791b371a3e4, State: Running, Role: LEADER
I20240502 01:53:09.158543 13367 consensus_queue.cc:237] T 00000000000000000000000000000000 P fad8aa713de7433db0f21791b371a3e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fad8aa713de7433db0f21791b371a3e4" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 44447 } }
I20240502 01:53:09.159221 13362 sys_catalog.cc:564] T 00000000000000000000000000000000 P fad8aa713de7433db0f21791b371a3e4 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 01:53:09.161381 13369 sys_catalog.cc:455] T 00000000000000000000000000000000 P fad8aa713de7433db0f21791b371a3e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fad8aa713de7433db0f21791b371a3e4. Latest consensus state: current_term: 1 leader_uuid: "fad8aa713de7433db0f21791b371a3e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fad8aa713de7433db0f21791b371a3e4" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 44447 } } }
I20240502 01:53:09.161783 13369 sys_catalog.cc:458] T 00000000000000000000000000000000 P fad8aa713de7433db0f21791b371a3e4 [sys.catalog]: This master's current role is: LEADER
I20240502 01:53:09.162508 13368 sys_catalog.cc:455] T 00000000000000000000000000000000 P fad8aa713de7433db0f21791b371a3e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fad8aa713de7433db0f21791b371a3e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fad8aa713de7433db0f21791b371a3e4" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 44447 } } }
I20240502 01:53:09.163309 13368 sys_catalog.cc:458] T 00000000000000000000000000000000 P fad8aa713de7433db0f21791b371a3e4 [sys.catalog]: This master's current role is: LEADER
I20240502 01:53:09.164110 13372 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 01:53:09.175755 13372 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 01:53:09.185310 13372 catalog_manager.cc:1347] Generated new cluster ID: be98629abebf42c98da8ad15ea882103
I20240502 01:53:09.185438 13372 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 01:53:09.208020 13372 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 01:53:09.209292 13372 catalog_manager.cc:1504] Loading token signing keys...
I20240502 01:53:09.219763 13372 catalog_manager.cc:5857] T 00000000000000000000000000000000 P fad8aa713de7433db0f21791b371a3e4: Generated new TSK 0
I20240502 01:53:09.220671 13372 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 01:53:09.227752  9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
/tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.193:0
--local_ip_for_outbound_sockets=127.9.176.193
--webserver_interface=127.9.176.193
--webserver_port=0
--tserver_master_addrs=127.9.176.254:44447
--builtin_ntp_servers=127.9.176.212:33337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 01:53:09.575309 13387 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:09.575682 13387 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:53:09.575829 13387 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:09.583729 13387 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:53:09.583863 13387 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:53:09.584024 13387 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.193
I20240502 01:53:09.593554 13387 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:33337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.9.176.193
--webserver_port=0
--tserver_master_addrs=127.9.176.254:44447
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.193
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:53:09.594942 13387 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:53:09.596809 13387 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:53:09.609411 13392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:09.610002 13393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:09.610558 13395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:53:09.611316 13387 server_base.cc:1034] running on GCE node
I20240502 01:53:09.612035 13387 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:53:09.613047 13387 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:53:09.614260 13387 hybrid_clock.cc:648] HybridClock initialized: now 1714614789614209 us; error 45 us; skew 500 ppm
I20240502 01:53:09.614574 13387 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:53:09.616926 13387 webserver.cc:460] Webserver started at http://127.9.176.193:38981/ using document root <none> and password file <none>
I20240502 01:53:09.617686 13387 fs_manager.cc:359] Metadata directory not provided
I20240502 01:53:09.617810 13387 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:53:09.618160 13387 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:53:09.620993 13387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "5359629f948641f4b56e35ad6b6a15c4"
format_stamp: "Formatted at 2024-05-02 01:53:09 on dist-test-slave-j3hh"
I20240502 01:53:09.621824 13387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "5359629f948641f4b56e35ad6b6a15c4"
format_stamp: "Formatted at 2024-05-02 01:53:09 on dist-test-slave-j3hh"
I20240502 01:53:09.627370 13387 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.006s	sys 0.001s
I20240502 01:53:09.631101 13401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:53:09.632630 13387 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240502 01:53:09.632807 13387 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "5359629f948641f4b56e35ad6b6a15c4"
format_stamp: "Formatted at 2024-05-02 01:53:09 on dist-test-slave-j3hh"
I20240502 01:53:09.633020 13387 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:53:09.671373 13387 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:53:09.672370 13387 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:53:09.672660 13387 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:53:09.673679 13387 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:53:09.675280 13387 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:53:09.675384 13387 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:53:09.675500 13387 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:53:09.675590 13387 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:53:09.709090 13387 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.193:33849
I20240502 01:53:09.709158 13516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.193:33849 every 8 connection(s)
I20240502 01:53:09.710443 13387 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 01:53:09.713274  9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu as pid 13387
I20240502 01:53:09.713534  9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 01:53:09.716683  9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
/tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.194:0
--local_ip_for_outbound_sockets=127.9.176.194
--webserver_interface=127.9.176.194
--webserver_port=0
--tserver_master_addrs=127.9.176.254:44447
--builtin_ntp_servers=127.9.176.212:33337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:53:09.724422 13517 heartbeater.cc:347] Connected to a master server at 127.9.176.254:44447
I20240502 01:53:09.724844 13517 heartbeater.cc:464] Registering TS with master...
I20240502 01:53:09.725821 13517 heartbeater.cc:511] Master 127.9.176.254:44447 requested a full tablet report, sending...
I20240502 01:53:09.728377 13326 ts_manager.cc:194] Registered new tserver with Master: 5359629f948641f4b56e35ad6b6a15c4 (127.9.176.193:33849)
I20240502 01:53:09.730727 13326 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.176.193:51527
W20240502 01:53:10.060600 13521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:10.060912 13521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:53:10.061012 13521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:10.068781 13521 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:53:10.068944 13521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:53:10.069063 13521 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.194
I20240502 01:53:10.078356 13521 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:33337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.9.176.194
--webserver_port=0
--tserver_master_addrs=127.9.176.254:44447
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.194
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:53:10.079746 13521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:53:10.081558 13521 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:53:10.094166 13527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:10.094759 13526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:10.095273 13529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:53:10.095970 13521 server_base.cc:1034] running on GCE node
I20240502 01:53:10.096592 13521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:53:10.097558 13521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:53:10.098766 13521 hybrid_clock.cc:648] HybridClock initialized: now 1714614790098715 us; error 46 us; skew 500 ppm
I20240502 01:53:10.099087 13521 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:53:10.101469 13521 webserver.cc:460] Webserver started at http://127.9.176.194:39417/ using document root <none> and password file <none>
I20240502 01:53:10.102259 13521 fs_manager.cc:359] Metadata directory not provided
I20240502 01:53:10.102393 13521 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:53:10.102710 13521 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:53:10.105458 13521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "8446b7a7ea67486288d174eea7a3b3c6"
format_stamp: "Formatted at 2024-05-02 01:53:10 on dist-test-slave-j3hh"
I20240502 01:53:10.106359 13521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "8446b7a7ea67486288d174eea7a3b3c6"
format_stamp: "Formatted at 2024-05-02 01:53:10 on dist-test-slave-j3hh"
I20240502 01:53:10.112023 13521 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.007s	sys 0.001s
I20240502 01:53:10.115795 13535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:53:10.117280 13521 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20240502 01:53:10.117462 13521 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "8446b7a7ea67486288d174eea7a3b3c6"
format_stamp: "Formatted at 2024-05-02 01:53:10 on dist-test-slave-j3hh"
I20240502 01:53:10.117673 13521 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:53:10.133466 13521 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:53:10.134431 13521 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:53:10.134725 13521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:53:10.135865 13521 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:53:10.137526 13521 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:53:10.137634 13521 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:53:10.137770 13521 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:53:10.137845 13521 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:53:10.170428 13521 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.194:34851
I20240502 01:53:10.170478 13649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.194:34851 every 8 connection(s)
I20240502 01:53:10.171916 13521 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 01:53:10.175384  9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu as pid 13521
I20240502 01:53:10.175588  9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 01:53:10.178769  9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
/tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.195:0
--local_ip_for_outbound_sockets=127.9.176.195
--webserver_interface=127.9.176.195
--webserver_port=0
--tserver_master_addrs=127.9.176.254:44447
--builtin_ntp_servers=127.9.176.212:33337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:53:10.186705 13650 heartbeater.cc:347] Connected to a master server at 127.9.176.254:44447
I20240502 01:53:10.187076 13650 heartbeater.cc:464] Registering TS with master...
I20240502 01:53:10.188015 13650 heartbeater.cc:511] Master 127.9.176.254:44447 requested a full tablet report, sending...
I20240502 01:53:10.189715 13326 ts_manager.cc:194] Registered new tserver with Master: 8446b7a7ea67486288d174eea7a3b3c6 (127.9.176.194:34851)
I20240502 01:53:10.190963 13326 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.176.194:38515
W20240502 01:53:10.536528 13653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:10.536849 13653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:53:10.536950 13653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:10.544605 13653 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:53:10.544744 13653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:53:10.544871 13653 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.195
I20240502 01:53:10.554117 13653 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:33337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.9.176.195
--webserver_port=0
--tserver_master_addrs=127.9.176.254:44447
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.195
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:53:10.555493 13653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:53:10.557338 13653 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:53:10.570050 13661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:10.570685 13659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:10.571290 13658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:53:10.572070 13653 server_base.cc:1034] running on GCE node
I20240502 01:53:10.572731 13653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:53:10.573757 13653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:53:10.574982 13653 hybrid_clock.cc:648] HybridClock initialized: now 1714614790574926 us; error 54 us; skew 500 ppm
I20240502 01:53:10.575330 13653 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:53:10.577929 13653 webserver.cc:460] Webserver started at http://127.9.176.195:40625/ using document root <none> and password file <none>
I20240502 01:53:10.578733 13653 fs_manager.cc:359] Metadata directory not provided
I20240502 01:53:10.578858 13653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:53:10.579190 13653 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:53:10.582021 13653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "b4abc88ed2e54830ae3a317629916117"
format_stamp: "Formatted at 2024-05-02 01:53:10 on dist-test-slave-j3hh"
I20240502 01:53:10.582868 13653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "b4abc88ed2e54830ae3a317629916117"
format_stamp: "Formatted at 2024-05-02 01:53:10 on dist-test-slave-j3hh"
I20240502 01:53:10.588593 13653 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.006s	sys 0.001s
I20240502 01:53:10.592468 13667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:53:10.593979 13653 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20240502 01:53:10.594158 13653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "b4abc88ed2e54830ae3a317629916117"
format_stamp: "Formatted at 2024-05-02 01:53:10 on dist-test-slave-j3hh"
I20240502 01:53:10.594383 13653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:53:10.626281 13653 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:53:10.627245 13653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:53:10.627548 13653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:53:10.628643 13653 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:53:10.630198 13653 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:53:10.630302 13653 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:53:10.630426 13653 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:53:10.630499 13653 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:53:10.663679 13653 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.195:41211
I20240502 01:53:10.663753 13781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.195:41211 every 8 connection(s)
I20240502 01:53:10.665069 13653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 01:53:10.666931  9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu as pid 13653
I20240502 01:53:10.667114  9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 01:53:10.670374  9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
/tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.196:0
--local_ip_for_outbound_sockets=127.9.176.196
--webserver_interface=127.9.176.196
--webserver_port=0
--tserver_master_addrs=127.9.176.254:44447
--builtin_ntp_servers=127.9.176.212:33337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:53:10.679311 13782 heartbeater.cc:347] Connected to a master server at 127.9.176.254:44447
I20240502 01:53:10.679708 13782 heartbeater.cc:464] Registering TS with master...
I20240502 01:53:10.680608 13782 heartbeater.cc:511] Master 127.9.176.254:44447 requested a full tablet report, sending...
I20240502 01:53:10.682201 13326 ts_manager.cc:194] Registered new tserver with Master: b4abc88ed2e54830ae3a317629916117 (127.9.176.195:41211)
I20240502 01:53:10.683037 13326 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.176.195:52721
I20240502 01:53:10.734062 13517 heartbeater.cc:503] Master 127.9.176.254:44447 was elected leader, sending a full tablet report...
W20240502 01:53:11.031179 13785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:11.031522 13785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:53:11.031679 13785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:11.039846 13785 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:53:11.040002 13785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:53:11.040206 13785 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.196
I20240502 01:53:11.049988 13785 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:33337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.9.176.196
--webserver_port=0
--tserver_master_addrs=127.9.176.254:44447
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.196
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:53:11.051399 13785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:53:11.053273 13785 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:53:11.066485 13790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:11.067077 13791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:11.067762 13793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:53:11.068470 13785 server_base.cc:1034] running on GCE node
I20240502 01:53:11.069145 13785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:53:11.070181 13785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:53:11.071411 13785 hybrid_clock.cc:648] HybridClock initialized: now 1714614791071361 us; error 41 us; skew 500 ppm
I20240502 01:53:11.071806 13785 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:53:11.074270 13785 webserver.cc:460] Webserver started at http://127.9.176.196:34361/ using document root <none> and password file <none>
I20240502 01:53:11.075098 13785 fs_manager.cc:359] Metadata directory not provided
I20240502 01:53:11.075227 13785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:53:11.075614 13785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:53:11.078606 13785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "ce90a4bd06ba41dd8e5aa227de9780a5"
format_stamp: "Formatted at 2024-05-02 01:53:11 on dist-test-slave-j3hh"
I20240502 01:53:11.079465 13785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "ce90a4bd06ba41dd8e5aa227de9780a5"
format_stamp: "Formatted at 2024-05-02 01:53:11 on dist-test-slave-j3hh"
I20240502 01:53:11.085412 13785 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.000s	sys 0.004s
I20240502 01:53:11.089322 13799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:53:11.090893 13785 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.000s
I20240502 01:53:11.091090 13785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "ce90a4bd06ba41dd8e5aa227de9780a5"
format_stamp: "Formatted at 2024-05-02 01:53:11 on dist-test-slave-j3hh"
I20240502 01:53:11.091334 13785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:53:11.156620 13785 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:53:11.157610 13785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:53:11.157939 13785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:53:11.159039 13785 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:53:11.160813 13785 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:53:11.160923 13785 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:53:11.161053 13785 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:53:11.161130 13785 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:53:11.193929 13650 heartbeater.cc:503] Master 127.9.176.254:44447 was elected leader, sending a full tablet report...
I20240502 01:53:11.196775 13785 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.196:41329
I20240502 01:53:11.196836 13913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.196:41329 every 8 connection(s)
I20240502 01:53:11.198191 13785 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 01:53:11.198940  9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu as pid 13785
I20240502 01:53:11.199118  9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 01:53:11.202778  9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
/tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.197:0
--local_ip_for_outbound_sockets=127.9.176.197
--webserver_interface=127.9.176.197
--webserver_port=0
--tserver_master_addrs=127.9.176.254:44447
--builtin_ntp_servers=127.9.176.212:33337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:53:11.215106 13914 heartbeater.cc:347] Connected to a master server at 127.9.176.254:44447
I20240502 01:53:11.215472 13914 heartbeater.cc:464] Registering TS with master...
I20240502 01:53:11.216406 13914 heartbeater.cc:511] Master 127.9.176.254:44447 requested a full tablet report, sending...
I20240502 01:53:11.218040 13326 ts_manager.cc:194] Registered new tserver with Master: ce90a4bd06ba41dd8e5aa227de9780a5 (127.9.176.196:41329)
I20240502 01:53:11.218879 13326 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.176.196:39803
W20240502 01:53:11.553001 13918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:11.553361 13918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:53:11.553515 13918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:11.561419 13918 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:53:11.561589 13918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:53:11.561765 13918 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.197
I20240502 01:53:11.571399 13918 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:33337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.197:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.9.176.197
--webserver_port=0
--tserver_master_addrs=127.9.176.254:44447
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.197
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:53:11.572850 13918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:53:11.574739 13918 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:53:11.588547 13924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:11.589247 13923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:11.589733 13927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:53:11.590273 13918 server_base.cc:1034] running on GCE node
I20240502 01:53:11.590917 13918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:53:11.591964 13918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:53:11.593183 13918 hybrid_clock.cc:648] HybridClock initialized: now 1714614791593126 us; error 54 us; skew 500 ppm
I20240502 01:53:11.593531 13918 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:53:11.600594 13918 webserver.cc:460] Webserver started at http://127.9.176.197:36841/ using document root <none> and password file <none>
I20240502 01:53:11.601438 13918 fs_manager.cc:359] Metadata directory not provided
I20240502 01:53:11.601570 13918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:53:11.601924 13918 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:53:11.607019 13918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "7e74a3d5df6a483c95c9dabfc99577bc"
format_stamp: "Formatted at 2024-05-02 01:53:11 on dist-test-slave-j3hh"
I20240502 01:53:11.607990 13918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "7e74a3d5df6a483c95c9dabfc99577bc"
format_stamp: "Formatted at 2024-05-02 01:53:11 on dist-test-slave-j3hh"
I20240502 01:53:11.614054 13918 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.000s	sys 0.005s
I20240502 01:53:11.617892 13933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:53:11.619406 13918 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.000s	sys 0.002s
I20240502 01:53:11.619630 13918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "7e74a3d5df6a483c95c9dabfc99577bc"
format_stamp: "Formatted at 2024-05-02 01:53:11 on dist-test-slave-j3hh"
I20240502 01:53:11.619882 13918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:53:11.642976 13918 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:53:11.644013 13918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:53:11.644312 13918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:53:11.645396 13918 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:53:11.647001 13918 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:53:11.647120 13918 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:53:11.647257 13918 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:53:11.647331 13918 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:53:11.681658 13918 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.197:35757
I20240502 01:53:11.681711 14047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.197:35757 every 8 connection(s)
I20240502 01:53:11.683068 13918 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 01:53:11.685704 13782 heartbeater.cc:503] Master 127.9.176.254:44447 was elected leader, sending a full tablet report...
I20240502 01:53:11.691891  9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu as pid 13918
I20240502 01:53:11.692126  9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 01:53:11.696015  9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
/tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.198:0
--local_ip_for_outbound_sockets=127.9.176.198
--webserver_interface=127.9.176.198
--webserver_port=0
--tserver_master_addrs=127.9.176.254:44447
--builtin_ntp_servers=127.9.176.212:33337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:53:11.696348 14049 heartbeater.cc:347] Connected to a master server at 127.9.176.254:44447
I20240502 01:53:11.696709 14049 heartbeater.cc:464] Registering TS with master...
I20240502 01:53:11.697690 14049 heartbeater.cc:511] Master 127.9.176.254:44447 requested a full tablet report, sending...
I20240502 01:53:11.699440 13326 ts_manager.cc:194] Registered new tserver with Master: 7e74a3d5df6a483c95c9dabfc99577bc (127.9.176.197:35757)
I20240502 01:53:11.700688 13326 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.176.197:32835
W20240502 01:53:12.048151 14052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:12.048472 14052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:53:12.048563 14052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:12.056324 14052 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:53:12.056473 14052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:53:12.056603 14052 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.198
I20240502 01:53:12.066176 14052 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:33337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.198:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.9.176.198
--webserver_port=0
--tserver_master_addrs=127.9.176.254:44447
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.198
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:53:12.067487 14052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:53:12.069370 14052 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:53:12.082546 14057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:12.083204 14058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:12.083588 14060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:53:12.084394 14052 server_base.cc:1034] running on GCE node
I20240502 01:53:12.085067 14052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:53:12.086161 14052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:53:12.087399 14052 hybrid_clock.cc:648] HybridClock initialized: now 1714614792087342 us; error 62 us; skew 500 ppm
I20240502 01:53:12.087791 14052 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:53:12.090282 14052 webserver.cc:460] Webserver started at http://127.9.176.198:37743/ using document root <none> and password file <none>
I20240502 01:53:12.091179 14052 fs_manager.cc:359] Metadata directory not provided
I20240502 01:53:12.091313 14052 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:53:12.091706 14052 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:53:12.094588 14052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f"
format_stamp: "Formatted at 2024-05-02 01:53:12 on dist-test-slave-j3hh"
I20240502 01:53:12.095444 14052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f"
format_stamp: "Formatted at 2024-05-02 01:53:12 on dist-test-slave-j3hh"
I20240502 01:53:12.101473 14052 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.005s	sys 0.000s
I20240502 01:53:12.105499 14066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:53:12.107077 14052 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20240502 01:53:12.107295 14052 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f"
format_stamp: "Formatted at 2024-05-02 01:53:12 on dist-test-slave-j3hh"
I20240502 01:53:12.107542 14052 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:53:12.139443 14052 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:53:12.140481 14052 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:53:12.140806 14052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:53:12.141878 14052 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:53:12.143508 14052 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:53:12.143644 14052 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.001s
I20240502 01:53:12.143777 14052 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:53:12.143854 14052 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:53:12.178300 14052 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.198:38527
I20240502 01:53:12.178359 14181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.198:38527 every 8 connection(s)
I20240502 01:53:12.179795 14052 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 01:53:12.183990  9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu as pid 14052
I20240502 01:53:12.184211  9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240502 01:53:12.187752  9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
/tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.199:0
--local_ip_for_outbound_sockets=127.9.176.199
--webserver_interface=127.9.176.199
--webserver_port=0
--tserver_master_addrs=127.9.176.254:44447
--builtin_ntp_servers=127.9.176.212:33337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:53:12.193698 14182 heartbeater.cc:347] Connected to a master server at 127.9.176.254:44447
I20240502 01:53:12.194027 14182 heartbeater.cc:464] Registering TS with master...
I20240502 01:53:12.194888 14182 heartbeater.cc:511] Master 127.9.176.254:44447 requested a full tablet report, sending...
I20240502 01:53:12.196697 13326 ts_manager.cc:194] Registered new tserver with Master: 9e4fa889fab047baa5bd3dcd77b2ca8f (127.9.176.198:38527)
I20240502 01:53:12.197917 13326 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.176.198:58453
I20240502 01:53:12.221849 13914 heartbeater.cc:503] Master 127.9.176.254:44447 was elected leader, sending a full tablet report...
W20240502 01:53:12.539901 14185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:12.540223 14185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:53:12.540333 14185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:12.547999 14185 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:53:12.548135 14185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:53:12.548265 14185 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.199
I20240502 01:53:12.557489 14185 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:33337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.199:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.9.176.199
--webserver_port=0
--tserver_master_addrs=127.9.176.254:44447
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.199
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:53:12.558861 14185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:53:12.560894 14185 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:53:12.573964 14190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:12.574587 14191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:12.575160 14193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:53:12.575984 14185 server_base.cc:1034] running on GCE node
I20240502 01:53:12.576702 14185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:53:12.577843 14185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:53:12.579057 14185 hybrid_clock.cc:648] HybridClock initialized: now 1714614792578992 us; error 59 us; skew 500 ppm
I20240502 01:53:12.579416 14185 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:53:12.581940 14185 webserver.cc:460] Webserver started at http://127.9.176.199:35747/ using document root <none> and password file <none>
I20240502 01:53:12.582758 14185 fs_manager.cc:359] Metadata directory not provided
I20240502 01:53:12.582885 14185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:53:12.583227 14185 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:53:12.586185 14185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "7505fefeaf804ed8aeb3672f386ab959"
format_stamp: "Formatted at 2024-05-02 01:53:12 on dist-test-slave-j3hh"
I20240502 01:53:12.587054 14185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "7505fefeaf804ed8aeb3672f386ab959"
format_stamp: "Formatted at 2024-05-02 01:53:12 on dist-test-slave-j3hh"
I20240502 01:53:12.592869 14185 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.004s	sys 0.004s
I20240502 01:53:12.596673 14199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:53:12.598227 14185 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.001s
I20240502 01:53:12.598405 14185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "7505fefeaf804ed8aeb3672f386ab959"
format_stamp: "Formatted at 2024-05-02 01:53:12 on dist-test-slave-j3hh"
I20240502 01:53:12.598623 14185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:53:12.631085 14185 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:53:12.632107 14185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:53:12.632406 14185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:53:12.633447 14185 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:53:12.635128 14185 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:53:12.635223 14185 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:53:12.635349 14185 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:53:12.635417 14185 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:53:12.673141 14185 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.199:38645
I20240502 01:53:12.673199 14313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.199:38645 every 8 connection(s)
I20240502 01:53:12.674566 14185 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 01:53:12.675704  9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu as pid 14185
I20240502 01:53:12.675925  9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240502 01:53:12.690990 14314 heartbeater.cc:347] Connected to a master server at 127.9.176.254:44447
I20240502 01:53:12.691365 14314 heartbeater.cc:464] Registering TS with master...
I20240502 01:53:12.692391 14314 heartbeater.cc:511] Master 127.9.176.254:44447 requested a full tablet report, sending...
I20240502 01:53:12.694094 13325 ts_manager.cc:194] Registered new tserver with Master: 7505fefeaf804ed8aeb3672f386ab959 (127.9.176.199:38645)
I20240502 01:53:12.695140 13325 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.176.199:48713
I20240502 01:53:12.703449 14049 heartbeater.cc:503] Master 127.9.176.254:44447 was elected leader, sending a full tablet report...
I20240502 01:53:12.706662  9923 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240502 01:53:12.724362 13325 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47160:
name: "TestTable"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
owner: "alice"
I20240502 01:53:12.764020 13848 tablet_service.cc:1467] Processing CreateTablet for tablet d38a05f7ffa94cc5abc50a76a83692b5 (DEFAULT_TABLE table=TestTable [id=821687150308482a8f601ee7300ef713]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:12.764017 13451 tablet_service.cc:1467] Processing CreateTablet for tablet d38a05f7ffa94cc5abc50a76a83692b5 (DEFAULT_TABLE table=TestTable [id=821687150308482a8f601ee7300ef713]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:12.766062 13848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d38a05f7ffa94cc5abc50a76a83692b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:12.766073 13451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d38a05f7ffa94cc5abc50a76a83692b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:12.778801 14330 tablet_bootstrap.cc:492] T d38a05f7ffa94cc5abc50a76a83692b5 P ce90a4bd06ba41dd8e5aa227de9780a5: Bootstrap starting.
I20240502 01:53:12.780232 14331 tablet_bootstrap.cc:492] T d38a05f7ffa94cc5abc50a76a83692b5 P 5359629f948641f4b56e35ad6b6a15c4: Bootstrap starting.
I20240502 01:53:12.783366 14331 tablet_bootstrap.cc:654] T d38a05f7ffa94cc5abc50a76a83692b5 P 5359629f948641f4b56e35ad6b6a15c4: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:12.783505 14330 tablet_bootstrap.cc:654] T d38a05f7ffa94cc5abc50a76a83692b5 P ce90a4bd06ba41dd8e5aa227de9780a5: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:12.784555 14331 log.cc:824] T d38a05f7ffa94cc5abc50a76a83692b5 P 5359629f948641f4b56e35ad6b6a15c4: Log is configured to *not* fsync() on all Append() calls
I20240502 01:53:12.784740 14330 log.cc:824] T d38a05f7ffa94cc5abc50a76a83692b5 P ce90a4bd06ba41dd8e5aa227de9780a5: Log is configured to *not* fsync() on all Append() calls
I20240502 01:53:12.787398 14331 tablet_bootstrap.cc:492] T d38a05f7ffa94cc5abc50a76a83692b5 P 5359629f948641f4b56e35ad6b6a15c4: No bootstrap required, opened a new log
I20240502 01:53:12.787410 14330 tablet_bootstrap.cc:492] T d38a05f7ffa94cc5abc50a76a83692b5 P ce90a4bd06ba41dd8e5aa227de9780a5: No bootstrap required, opened a new log
I20240502 01:53:12.787711 14331 ts_tablet_manager.cc:1367] T d38a05f7ffa94cc5abc50a76a83692b5 P 5359629f948641f4b56e35ad6b6a15c4: Time spent bootstrapping tablet: real 0.008s	user 0.007s	sys 0.000s
I20240502 01:53:12.787727 14330 ts_tablet_manager.cc:1367] T d38a05f7ffa94cc5abc50a76a83692b5 P ce90a4bd06ba41dd8e5aa227de9780a5: Time spent bootstrapping tablet: real 0.009s	user 0.007s	sys 0.000s
I20240502 01:53:12.794651 14331 raft_consensus.cc:348] T d38a05f7ffa94cc5abc50a76a83692b5 P 5359629f948641f4b56e35ad6b6a15c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 41329 } }
I20240502 01:53:12.794713 14330 raft_consensus.cc:348] T d38a05f7ffa94cc5abc50a76a83692b5 P ce90a4bd06ba41dd8e5aa227de9780a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 41329 } }
I20240502 01:53:12.795083 14330 raft_consensus.cc:374] T d38a05f7ffa94cc5abc50a76a83692b5 P ce90a4bd06ba41dd8e5aa227de9780a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:12.795097 14331 raft_consensus.cc:374] T d38a05f7ffa94cc5abc50a76a83692b5 P 5359629f948641f4b56e35ad6b6a15c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:12.795214 14331 raft_consensus.cc:729] T d38a05f7ffa94cc5abc50a76a83692b5 P 5359629f948641f4b56e35ad6b6a15c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5359629f948641f4b56e35ad6b6a15c4, State: Initialized, Role: FOLLOWER
I20240502 01:53:12.795214 14330 raft_consensus.cc:729] T d38a05f7ffa94cc5abc50a76a83692b5 P ce90a4bd06ba41dd8e5aa227de9780a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce90a4bd06ba41dd8e5aa227de9780a5, State: Initialized, Role: FOLLOWER
I20240502 01:53:12.796015 14331 consensus_queue.cc:260] T d38a05f7ffa94cc5abc50a76a83692b5 P 5359629f948641f4b56e35ad6b6a15c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 41329 } }
I20240502 01:53:12.796041 14330 consensus_queue.cc:260] T d38a05f7ffa94cc5abc50a76a83692b5 P ce90a4bd06ba41dd8e5aa227de9780a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 41329 } }
I20240502 01:53:12.798061 14330 ts_tablet_manager.cc:1398] T d38a05f7ffa94cc5abc50a76a83692b5 P ce90a4bd06ba41dd8e5aa227de9780a5: Time spent starting tablet: real 0.010s	user 0.009s	sys 0.000s
I20240502 01:53:12.799401 14331 ts_tablet_manager.cc:1398] T d38a05f7ffa94cc5abc50a76a83692b5 P 5359629f948641f4b56e35ad6b6a15c4: Time spent starting tablet: real 0.012s	user 0.010s	sys 0.000s
I20240502 01:53:12.853508 14335 raft_consensus.cc:482] T d38a05f7ffa94cc5abc50a76a83692b5 P ce90a4bd06ba41dd8e5aa227de9780a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:12.853789 14335 raft_consensus.cc:504] T d38a05f7ffa94cc5abc50a76a83692b5 P ce90a4bd06ba41dd8e5aa227de9780a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 41329 } }
I20240502 01:53:12.855353 14335 leader_election.cc:288] T d38a05f7ffa94cc5abc50a76a83692b5 P ce90a4bd06ba41dd8e5aa227de9780a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5359629f948641f4b56e35ad6b6a15c4 (127.9.176.193:33849)
I20240502 01:53:12.862237 13471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d38a05f7ffa94cc5abc50a76a83692b5" candidate_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5359629f948641f4b56e35ad6b6a15c4" is_pre_election: true
I20240502 01:53:12.862744 13471 raft_consensus.cc:2454] T d38a05f7ffa94cc5abc50a76a83692b5 P 5359629f948641f4b56e35ad6b6a15c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce90a4bd06ba41dd8e5aa227de9780a5 in term 0.
I20240502 01:53:12.863386 13804 leader_election.cc:302] T d38a05f7ffa94cc5abc50a76a83692b5 P ce90a4bd06ba41dd8e5aa227de9780a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 5359629f948641f4b56e35ad6b6a15c4, ce90a4bd06ba41dd8e5aa227de9780a5; no voters: 
I20240502 01:53:12.863837 14335 raft_consensus.cc:2781] T d38a05f7ffa94cc5abc50a76a83692b5 P ce90a4bd06ba41dd8e5aa227de9780a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:12.863986 14335 raft_consensus.cc:482] T d38a05f7ffa94cc5abc50a76a83692b5 P ce90a4bd06ba41dd8e5aa227de9780a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:12.864101 14335 raft_consensus.cc:3037] T d38a05f7ffa94cc5abc50a76a83692b5 P ce90a4bd06ba41dd8e5aa227de9780a5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:12.866359 14335 raft_consensus.cc:504] T d38a05f7ffa94cc5abc50a76a83692b5 P ce90a4bd06ba41dd8e5aa227de9780a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 41329 } }
I20240502 01:53:12.866902 14335 leader_election.cc:288] T d38a05f7ffa94cc5abc50a76a83692b5 P ce90a4bd06ba41dd8e5aa227de9780a5 [CANDIDATE]: Term 1 election: Requested vote from peers 5359629f948641f4b56e35ad6b6a15c4 (127.9.176.193:33849)
I20240502 01:53:12.867496 13471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d38a05f7ffa94cc5abc50a76a83692b5" candidate_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5359629f948641f4b56e35ad6b6a15c4"
I20240502 01:53:12.867727 13471 raft_consensus.cc:3037] T d38a05f7ffa94cc5abc50a76a83692b5 P 5359629f948641f4b56e35ad6b6a15c4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:12.869917 13471 raft_consensus.cc:2454] T d38a05f7ffa94cc5abc50a76a83692b5 P 5359629f948641f4b56e35ad6b6a15c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce90a4bd06ba41dd8e5aa227de9780a5 in term 1.
I20240502 01:53:12.870368 13804 leader_election.cc:302] T d38a05f7ffa94cc5abc50a76a83692b5 P ce90a4bd06ba41dd8e5aa227de9780a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 5359629f948641f4b56e35ad6b6a15c4, ce90a4bd06ba41dd8e5aa227de9780a5; no voters: 
I20240502 01:53:12.870697 14335 raft_consensus.cc:2781] T d38a05f7ffa94cc5abc50a76a83692b5 P ce90a4bd06ba41dd8e5aa227de9780a5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:12.871160 14335 raft_consensus.cc:686] T d38a05f7ffa94cc5abc50a76a83692b5 P ce90a4bd06ba41dd8e5aa227de9780a5 [term 1 LEADER]: Becoming Leader. State: Replica: ce90a4bd06ba41dd8e5aa227de9780a5, State: Running, Role: LEADER
I20240502 01:53:12.871652 14335 consensus_queue.cc:237] T d38a05f7ffa94cc5abc50a76a83692b5 P ce90a4bd06ba41dd8e5aa227de9780a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 41329 } }
I20240502 01:53:12.876646 13326 catalog_manager.cc:5484] T d38a05f7ffa94cc5abc50a76a83692b5 P ce90a4bd06ba41dd8e5aa227de9780a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to ce90a4bd06ba41dd8e5aa227de9780a5 (127.9.176.196). New cstate: current_term: 1 leader_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 41329 } health_report { overall_health: HEALTHY } } }
I20240502 01:53:12.910594  9923 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240502 01:53:12.912958  9923 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 5359629f948641f4b56e35ad6b6a15c4 to finish bootstrapping
I20240502 01:53:12.921630  9923 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 8446b7a7ea67486288d174eea7a3b3c6 to finish bootstrapping
I20240502 01:53:12.931403  9923 ts_itest-base.cc:233] Waiting for 0 tablets on tserver b4abc88ed2e54830ae3a317629916117 to finish bootstrapping
I20240502 01:53:12.941078  9923 ts_itest-base.cc:233] Waiting for 1 tablets on tserver ce90a4bd06ba41dd8e5aa227de9780a5 to finish bootstrapping
I20240502 01:53:12.949694  9923 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 7e74a3d5df6a483c95c9dabfc99577bc to finish bootstrapping
W20240502 01:53:12.952378 13915 tablet.cc:2369] T d38a05f7ffa94cc5abc50a76a83692b5 P ce90a4bd06ba41dd8e5aa227de9780a5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:53:12.959355  9923 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 9e4fa889fab047baa5bd3dcd77b2ca8f to finish bootstrapping
W20240502 01:53:12.964625 13518 tablet.cc:2369] T d38a05f7ffa94cc5abc50a76a83692b5 P 5359629f948641f4b56e35ad6b6a15c4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:53:12.968729  9923 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 7505fefeaf804ed8aeb3672f386ab959 to finish bootstrapping
I20240502 01:53:12.978204  9923 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu pid 13785
I20240502 01:53:12.979300 13909 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:53:13.069710  9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu with pid 13785
I20240502 01:53:13.088313  9923 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu pid 13918
I20240502 01:53:13.089538 14043 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:53:13.172905  9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu with pid 13918
I20240502 01:53:13.194358  9923 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu pid 14052
I20240502 01:53:13.195673 14176 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:53:13.278837 14182 heartbeater.cc:503] Master 127.9.176.254:44447 was elected leader, sending a full tablet report...
I20240502 01:53:13.280030  9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu with pid 14052
W20240502 01:53:13.297396 13312 connection.cc:535] server connection from 127.9.176.198:58453 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 01:53:13.297955  9923 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu pid 14185
I20240502 01:53:13.299046 14309 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:53:13.382092  9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu with pid 14185
I20240502 01:53:14.685554 14350 raft_consensus.cc:482] T d38a05f7ffa94cc5abc50a76a83692b5 P 5359629f948641f4b56e35ad6b6a15c4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:14.685778 14350 raft_consensus.cc:504] T d38a05f7ffa94cc5abc50a76a83692b5 P 5359629f948641f4b56e35ad6b6a15c4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 41329 } }
I20240502 01:53:14.687304 14350 leader_election.cc:288] T d38a05f7ffa94cc5abc50a76a83692b5 P 5359629f948641f4b56e35ad6b6a15c4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ce90a4bd06ba41dd8e5aa227de9780a5 (127.9.176.196:41329)
W20240502 01:53:14.688737 13402 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.176.196:41329: connect: Connection refused (error 111)
W20240502 01:53:14.693270 13402 leader_election.cc:334] T d38a05f7ffa94cc5abc50a76a83692b5 P 5359629f948641f4b56e35ad6b6a15c4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ce90a4bd06ba41dd8e5aa227de9780a5 (127.9.176.196:41329): Network error: Client connection negotiation failed: client connection to 127.9.176.196:41329: connect: Connection refused (error 111)
I20240502 01:53:14.693460 13402 leader_election.cc:302] T d38a05f7ffa94cc5abc50a76a83692b5 P 5359629f948641f4b56e35ad6b6a15c4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 5359629f948641f4b56e35ad6b6a15c4; no voters: ce90a4bd06ba41dd8e5aa227de9780a5
I20240502 01:53:14.693895 14350 raft_consensus.cc:2726] T d38a05f7ffa94cc5abc50a76a83692b5 P 5359629f948641f4b56e35ad6b6a15c4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:53:16.612138 14353 raft_consensus.cc:482] T d38a05f7ffa94cc5abc50a76a83692b5 P 5359629f948641f4b56e35ad6b6a15c4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:16.612357 14353 raft_consensus.cc:504] T d38a05f7ffa94cc5abc50a76a83692b5 P 5359629f948641f4b56e35ad6b6a15c4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 41329 } }
I20240502 01:53:16.612967 14353 leader_election.cc:288] T d38a05f7ffa94cc5abc50a76a83692b5 P 5359629f948641f4b56e35ad6b6a15c4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ce90a4bd06ba41dd8e5aa227de9780a5 (127.9.176.196:41329)
W20240502 01:53:16.616568 13402 leader_election.cc:334] T d38a05f7ffa94cc5abc50a76a83692b5 P 5359629f948641f4b56e35ad6b6a15c4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ce90a4bd06ba41dd8e5aa227de9780a5 (127.9.176.196:41329): Network error: Client connection negotiation failed: client connection to 127.9.176.196:41329: connect: Connection refused (error 111)
I20240502 01:53:16.616746 13402 leader_election.cc:302] T d38a05f7ffa94cc5abc50a76a83692b5 P 5359629f948641f4b56e35ad6b6a15c4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 5359629f948641f4b56e35ad6b6a15c4; no voters: ce90a4bd06ba41dd8e5aa227de9780a5
I20240502 01:53:16.617034 14353 raft_consensus.cc:2726] T d38a05f7ffa94cc5abc50a76a83692b5 P 5359629f948641f4b56e35ad6b6a15c4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:53:17.150710 13326 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47160:
name: "rebalance_test_table_0"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 01:53:17.164269 13451 tablet_service.cc:1467] Processing CreateTablet for tablet 01fb93c5dab3401fa5a103ead4dd9708 (DEFAULT_TABLE table=rebalance_test_table_0 [id=42b769e9abfb4c36b2891f39a41750d2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:17.164981 13451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01fb93c5dab3401fa5a103ead4dd9708. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:17.180099 14364 tablet_bootstrap.cc:492] T 01fb93c5dab3401fa5a103ead4dd9708 P 5359629f948641f4b56e35ad6b6a15c4: Bootstrap starting.
I20240502 01:53:17.184448 14364 tablet_bootstrap.cc:654] T 01fb93c5dab3401fa5a103ead4dd9708 P 5359629f948641f4b56e35ad6b6a15c4: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:17.190896 13714 tablet_service.cc:1467] Processing CreateTablet for tablet d4511dd1019c4bdcbf9fc7e4bfe2afb7 (DEFAULT_TABLE table=rebalance_test_table_0 [id=42b769e9abfb4c36b2891f39a41750d2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:17.190843 13715 tablet_service.cc:1467] Processing CreateTablet for tablet daed3123a96a496da83a6110e6943264 (DEFAULT_TABLE table=rebalance_test_table_0 [id=42b769e9abfb4c36b2891f39a41750d2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:17.192728 13714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4511dd1019c4bdcbf9fc7e4bfe2afb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:17.192312 13582 tablet_service.cc:1467] Processing CreateTablet for tablet 01fb93c5dab3401fa5a103ead4dd9708 (DEFAULT_TABLE table=rebalance_test_table_0 [id=42b769e9abfb4c36b2891f39a41750d2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:17.192270 13584 tablet_service.cc:1467] Processing CreateTablet for tablet daed3123a96a496da83a6110e6943264 (DEFAULT_TABLE table=rebalance_test_table_0 [id=42b769e9abfb4c36b2891f39a41750d2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:17.192704 13583 tablet_service.cc:1467] Processing CreateTablet for tablet d4511dd1019c4bdcbf9fc7e4bfe2afb7 (DEFAULT_TABLE table=rebalance_test_table_0 [id=42b769e9abfb4c36b2891f39a41750d2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:17.193761 13715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet daed3123a96a496da83a6110e6943264. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:17.194067 13583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4511dd1019c4bdcbf9fc7e4bfe2afb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:17.195369 13582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01fb93c5dab3401fa5a103ead4dd9708. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:17.196441 14364 tablet_bootstrap.cc:492] T 01fb93c5dab3401fa5a103ead4dd9708 P 5359629f948641f4b56e35ad6b6a15c4: No bootstrap required, opened a new log
I20240502 01:53:17.196681 14364 ts_tablet_manager.cc:1367] T 01fb93c5dab3401fa5a103ead4dd9708 P 5359629f948641f4b56e35ad6b6a15c4: Time spent bootstrapping tablet: real 0.017s	user 0.005s	sys 0.000s
I20240502 01:53:17.197782 14364 raft_consensus.cc:348] T 01fb93c5dab3401fa5a103ead4dd9708 P 5359629f948641f4b56e35ad6b6a15c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } }
I20240502 01:53:17.198110 14364 raft_consensus.cc:374] T 01fb93c5dab3401fa5a103ead4dd9708 P 5359629f948641f4b56e35ad6b6a15c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:17.198256 14364 raft_consensus.cc:729] T 01fb93c5dab3401fa5a103ead4dd9708 P 5359629f948641f4b56e35ad6b6a15c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5359629f948641f4b56e35ad6b6a15c4, State: Initialized, Role: FOLLOWER
I20240502 01:53:17.198572 14364 consensus_queue.cc:260] T 01fb93c5dab3401fa5a103ead4dd9708 P 5359629f948641f4b56e35ad6b6a15c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } }
I20240502 01:53:17.198683 13584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet daed3123a96a496da83a6110e6943264. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:17.211205 14364 ts_tablet_manager.cc:1398] T 01fb93c5dab3401fa5a103ead4dd9708 P 5359629f948641f4b56e35ad6b6a15c4: Time spent starting tablet: real 0.014s	user 0.004s	sys 0.009s
I20240502 01:53:17.216516 14368 tablet_bootstrap.cc:492] T d4511dd1019c4bdcbf9fc7e4bfe2afb7 P 8446b7a7ea67486288d174eea7a3b3c6: Bootstrap starting.
I20240502 01:53:17.221174 14367 tablet_bootstrap.cc:492] T d4511dd1019c4bdcbf9fc7e4bfe2afb7 P b4abc88ed2e54830ae3a317629916117: Bootstrap starting.
I20240502 01:53:17.221400 14368 tablet_bootstrap.cc:654] T d4511dd1019c4bdcbf9fc7e4bfe2afb7 P 8446b7a7ea67486288d174eea7a3b3c6: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:17.223102 14368 log.cc:824] T d4511dd1019c4bdcbf9fc7e4bfe2afb7 P 8446b7a7ea67486288d174eea7a3b3c6: Log is configured to *not* fsync() on all Append() calls
I20240502 01:53:17.225008 14367 tablet_bootstrap.cc:654] T d4511dd1019c4bdcbf9fc7e4bfe2afb7 P b4abc88ed2e54830ae3a317629916117: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:17.226377 14367 log.cc:824] T d4511dd1019c4bdcbf9fc7e4bfe2afb7 P b4abc88ed2e54830ae3a317629916117: Log is configured to *not* fsync() on all Append() calls
I20240502 01:53:17.226450 14368 tablet_bootstrap.cc:492] T d4511dd1019c4bdcbf9fc7e4bfe2afb7 P 8446b7a7ea67486288d174eea7a3b3c6: No bootstrap required, opened a new log
I20240502 01:53:17.226742 14368 ts_tablet_manager.cc:1367] T d4511dd1019c4bdcbf9fc7e4bfe2afb7 P 8446b7a7ea67486288d174eea7a3b3c6: Time spent bootstrapping tablet: real 0.011s	user 0.007s	sys 0.000s
I20240502 01:53:17.229429 14367 tablet_bootstrap.cc:492] T d4511dd1019c4bdcbf9fc7e4bfe2afb7 P b4abc88ed2e54830ae3a317629916117: No bootstrap required, opened a new log
I20240502 01:53:17.229736 14367 ts_tablet_manager.cc:1367] T d4511dd1019c4bdcbf9fc7e4bfe2afb7 P b4abc88ed2e54830ae3a317629916117: Time spent bootstrapping tablet: real 0.009s	user 0.001s	sys 0.006s
I20240502 01:53:17.236078 14368 raft_consensus.cc:348] T d4511dd1019c4bdcbf9fc7e4bfe2afb7 P 8446b7a7ea67486288d174eea7a3b3c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } }
I20240502 01:53:17.236555 14368 raft_consensus.cc:374] T d4511dd1019c4bdcbf9fc7e4bfe2afb7 P 8446b7a7ea67486288d174eea7a3b3c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:17.236675 14368 raft_consensus.cc:729] T d4511dd1019c4bdcbf9fc7e4bfe2afb7 P 8446b7a7ea67486288d174eea7a3b3c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8446b7a7ea67486288d174eea7a3b3c6, State: Initialized, Role: FOLLOWER
I20240502 01:53:17.237505 14368 consensus_queue.cc:260] T d4511dd1019c4bdcbf9fc7e4bfe2afb7 P 8446b7a7ea67486288d174eea7a3b3c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } }
I20240502 01:53:17.238792 14367 raft_consensus.cc:348] T d4511dd1019c4bdcbf9fc7e4bfe2afb7 P b4abc88ed2e54830ae3a317629916117 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } }
I20240502 01:53:17.239261 14367 raft_consensus.cc:374] T d4511dd1019c4bdcbf9fc7e4bfe2afb7 P b4abc88ed2e54830ae3a317629916117 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:17.239414 14368 ts_tablet_manager.cc:1398] T d4511dd1019c4bdcbf9fc7e4bfe2afb7 P 8446b7a7ea67486288d174eea7a3b3c6: Time spent starting tablet: real 0.013s	user 0.010s	sys 0.004s
I20240502 01:53:17.239398 14367 raft_consensus.cc:729] T d4511dd1019c4bdcbf9fc7e4bfe2afb7 P b4abc88ed2e54830ae3a317629916117 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4abc88ed2e54830ae3a317629916117, State: Initialized, Role: FOLLOWER
I20240502 01:53:17.239899 14368 tablet_bootstrap.cc:492] T daed3123a96a496da83a6110e6943264 P 8446b7a7ea67486288d174eea7a3b3c6: Bootstrap starting.
I20240502 01:53:17.240190 14367 consensus_queue.cc:260] T d4511dd1019c4bdcbf9fc7e4bfe2afb7 P b4abc88ed2e54830ae3a317629916117 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } }
I20240502 01:53:17.241792 14368 tablet_bootstrap.cc:654] T daed3123a96a496da83a6110e6943264 P 8446b7a7ea67486288d174eea7a3b3c6: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:17.242436 14367 ts_tablet_manager.cc:1398] T d4511dd1019c4bdcbf9fc7e4bfe2afb7 P b4abc88ed2e54830ae3a317629916117: Time spent starting tablet: real 0.013s	user 0.014s	sys 0.000s
I20240502 01:53:17.243242 14367 tablet_bootstrap.cc:492] T daed3123a96a496da83a6110e6943264 P b4abc88ed2e54830ae3a317629916117: Bootstrap starting.
I20240502 01:53:17.245863 14367 tablet_bootstrap.cc:654] T daed3123a96a496da83a6110e6943264 P b4abc88ed2e54830ae3a317629916117: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:17.247303 14368 tablet_bootstrap.cc:492] T daed3123a96a496da83a6110e6943264 P 8446b7a7ea67486288d174eea7a3b3c6: No bootstrap required, opened a new log
I20240502 01:53:17.247462 14368 ts_tablet_manager.cc:1367] T daed3123a96a496da83a6110e6943264 P 8446b7a7ea67486288d174eea7a3b3c6: Time spent bootstrapping tablet: real 0.008s	user 0.005s	sys 0.000s
I20240502 01:53:17.248162 14367 tablet_bootstrap.cc:492] T daed3123a96a496da83a6110e6943264 P b4abc88ed2e54830ae3a317629916117: No bootstrap required, opened a new log
I20240502 01:53:17.248309 14367 ts_tablet_manager.cc:1367] T daed3123a96a496da83a6110e6943264 P b4abc88ed2e54830ae3a317629916117: Time spent bootstrapping tablet: real 0.005s	user 0.003s	sys 0.000s
I20240502 01:53:17.248562 14368 raft_consensus.cc:348] T daed3123a96a496da83a6110e6943264 P 8446b7a7ea67486288d174eea7a3b3c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } }
I20240502 01:53:17.248768 14368 raft_consensus.cc:374] T daed3123a96a496da83a6110e6943264 P 8446b7a7ea67486288d174eea7a3b3c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:17.248872 14368 raft_consensus.cc:729] T daed3123a96a496da83a6110e6943264 P 8446b7a7ea67486288d174eea7a3b3c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8446b7a7ea67486288d174eea7a3b3c6, State: Initialized, Role: FOLLOWER
I20240502 01:53:17.249136 14368 consensus_queue.cc:260] T daed3123a96a496da83a6110e6943264 P 8446b7a7ea67486288d174eea7a3b3c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } }
I20240502 01:53:17.249307 14367 raft_consensus.cc:348] T daed3123a96a496da83a6110e6943264 P b4abc88ed2e54830ae3a317629916117 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } }
I20240502 01:53:17.249699 14368 ts_tablet_manager.cc:1398] T daed3123a96a496da83a6110e6943264 P 8446b7a7ea67486288d174eea7a3b3c6: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.000s
I20240502 01:53:17.249974 14368 tablet_bootstrap.cc:492] T 01fb93c5dab3401fa5a103ead4dd9708 P 8446b7a7ea67486288d174eea7a3b3c6: Bootstrap starting.
I20240502 01:53:17.250059 14367 raft_consensus.cc:374] T daed3123a96a496da83a6110e6943264 P b4abc88ed2e54830ae3a317629916117 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:17.250180 14367 raft_consensus.cc:729] T daed3123a96a496da83a6110e6943264 P b4abc88ed2e54830ae3a317629916117 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4abc88ed2e54830ae3a317629916117, State: Initialized, Role: FOLLOWER
I20240502 01:53:17.250454 14367 consensus_queue.cc:260] T daed3123a96a496da83a6110e6943264 P b4abc88ed2e54830ae3a317629916117 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } }
I20240502 01:53:17.251374 14367 ts_tablet_manager.cc:1398] T daed3123a96a496da83a6110e6943264 P b4abc88ed2e54830ae3a317629916117: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.002s
I20240502 01:53:17.251873 14368 tablet_bootstrap.cc:654] T 01fb93c5dab3401fa5a103ead4dd9708 P 8446b7a7ea67486288d174eea7a3b3c6: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:17.254971 14368 tablet_bootstrap.cc:492] T 01fb93c5dab3401fa5a103ead4dd9708 P 8446b7a7ea67486288d174eea7a3b3c6: No bootstrap required, opened a new log
I20240502 01:53:17.255141 14368 ts_tablet_manager.cc:1367] T 01fb93c5dab3401fa5a103ead4dd9708 P 8446b7a7ea67486288d174eea7a3b3c6: Time spent bootstrapping tablet: real 0.005s	user 0.003s	sys 0.000s
I20240502 01:53:17.255834 14368 raft_consensus.cc:348] T 01fb93c5dab3401fa5a103ead4dd9708 P 8446b7a7ea67486288d174eea7a3b3c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } }
I20240502 01:53:17.256014 14368 raft_consensus.cc:374] T 01fb93c5dab3401fa5a103ead4dd9708 P 8446b7a7ea67486288d174eea7a3b3c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:17.256094 14368 raft_consensus.cc:729] T 01fb93c5dab3401fa5a103ead4dd9708 P 8446b7a7ea67486288d174eea7a3b3c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8446b7a7ea67486288d174eea7a3b3c6, State: Initialized, Role: FOLLOWER
I20240502 01:53:17.256338 14368 consensus_queue.cc:260] T 01fb93c5dab3401fa5a103ead4dd9708 P 8446b7a7ea67486288d174eea7a3b3c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } }
I20240502 01:53:17.256788 14368 ts_tablet_manager.cc:1398] T 01fb93c5dab3401fa5a103ead4dd9708 P 8446b7a7ea67486288d174eea7a3b3c6: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240502 01:53:17.289112 14372 raft_consensus.cc:482] T daed3123a96a496da83a6110e6943264 P b4abc88ed2e54830ae3a317629916117 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:17.289391 14372 raft_consensus.cc:504] T daed3123a96a496da83a6110e6943264 P b4abc88ed2e54830ae3a317629916117 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } }
I20240502 01:53:17.291016 14372 leader_election.cc:288] T daed3123a96a496da83a6110e6943264 P b4abc88ed2e54830ae3a317629916117 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8446b7a7ea67486288d174eea7a3b3c6 (127.9.176.194:34851)
I20240502 01:53:17.297406 13604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daed3123a96a496da83a6110e6943264" candidate_uuid: "b4abc88ed2e54830ae3a317629916117" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8446b7a7ea67486288d174eea7a3b3c6" is_pre_election: true
I20240502 01:53:17.297979 13604 raft_consensus.cc:2454] T daed3123a96a496da83a6110e6943264 P 8446b7a7ea67486288d174eea7a3b3c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4abc88ed2e54830ae3a317629916117 in term 0.
I20240502 01:53:17.298620 13668 leader_election.cc:302] T daed3123a96a496da83a6110e6943264 P b4abc88ed2e54830ae3a317629916117 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 8446b7a7ea67486288d174eea7a3b3c6, b4abc88ed2e54830ae3a317629916117; no voters: 
I20240502 01:53:17.299029 14372 raft_consensus.cc:2781] T daed3123a96a496da83a6110e6943264 P b4abc88ed2e54830ae3a317629916117 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:17.299155 14372 raft_consensus.cc:482] T daed3123a96a496da83a6110e6943264 P b4abc88ed2e54830ae3a317629916117 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:17.299257 14372 raft_consensus.cc:3037] T daed3123a96a496da83a6110e6943264 P b4abc88ed2e54830ae3a317629916117 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:17.301393 14372 raft_consensus.cc:504] T daed3123a96a496da83a6110e6943264 P b4abc88ed2e54830ae3a317629916117 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } }
I20240502 01:53:17.301946 14372 leader_election.cc:288] T daed3123a96a496da83a6110e6943264 P b4abc88ed2e54830ae3a317629916117 [CANDIDATE]: Term 1 election: Requested vote from peers 8446b7a7ea67486288d174eea7a3b3c6 (127.9.176.194:34851)
I20240502 01:53:17.302541 13604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daed3123a96a496da83a6110e6943264" candidate_uuid: "b4abc88ed2e54830ae3a317629916117" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8446b7a7ea67486288d174eea7a3b3c6"
I20240502 01:53:17.302731 13604 raft_consensus.cc:3037] T daed3123a96a496da83a6110e6943264 P 8446b7a7ea67486288d174eea7a3b3c6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:17.305006 13604 raft_consensus.cc:2454] T daed3123a96a496da83a6110e6943264 P 8446b7a7ea67486288d174eea7a3b3c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4abc88ed2e54830ae3a317629916117 in term 1.
I20240502 01:53:17.305502 13668 leader_election.cc:302] T daed3123a96a496da83a6110e6943264 P b4abc88ed2e54830ae3a317629916117 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 8446b7a7ea67486288d174eea7a3b3c6, b4abc88ed2e54830ae3a317629916117; no voters: 
I20240502 01:53:17.305851 14372 raft_consensus.cc:2781] T daed3123a96a496da83a6110e6943264 P b4abc88ed2e54830ae3a317629916117 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:17.306280 14372 raft_consensus.cc:686] T daed3123a96a496da83a6110e6943264 P b4abc88ed2e54830ae3a317629916117 [term 1 LEADER]: Becoming Leader. State: Replica: b4abc88ed2e54830ae3a317629916117, State: Running, Role: LEADER
I20240502 01:53:17.306826 14372 consensus_queue.cc:237] T daed3123a96a496da83a6110e6943264 P b4abc88ed2e54830ae3a317629916117 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } }
I20240502 01:53:17.311311 13326 catalog_manager.cc:5484] T daed3123a96a496da83a6110e6943264 P b4abc88ed2e54830ae3a317629916117 reported cstate change: term changed from 0 to 1, leader changed from <none> to b4abc88ed2e54830ae3a317629916117 (127.9.176.195). New cstate: current_term: 1 leader_uuid: "b4abc88ed2e54830ae3a317629916117" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } health_report { overall_health: HEALTHY } } }
I20240502 01:53:17.319173 14366 raft_consensus.cc:482] T 01fb93c5dab3401fa5a103ead4dd9708 P 5359629f948641f4b56e35ad6b6a15c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:17.319427 14366 raft_consensus.cc:504] T 01fb93c5dab3401fa5a103ead4dd9708 P 5359629f948641f4b56e35ad6b6a15c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } }
I20240502 01:53:17.320304 14366 leader_election.cc:288] T 01fb93c5dab3401fa5a103ead4dd9708 P 5359629f948641f4b56e35ad6b6a15c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8446b7a7ea67486288d174eea7a3b3c6 (127.9.176.194:34851)
I20240502 01:53:17.328089 13604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01fb93c5dab3401fa5a103ead4dd9708" candidate_uuid: "5359629f948641f4b56e35ad6b6a15c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8446b7a7ea67486288d174eea7a3b3c6" is_pre_election: true
I20240502 01:53:17.328342 13604 raft_consensus.cc:2454] T 01fb93c5dab3401fa5a103ead4dd9708 P 8446b7a7ea67486288d174eea7a3b3c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5359629f948641f4b56e35ad6b6a15c4 in term 0.
I20240502 01:53:17.328744 13402 leader_election.cc:302] T 01fb93c5dab3401fa5a103ead4dd9708 P 5359629f948641f4b56e35ad6b6a15c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 5359629f948641f4b56e35ad6b6a15c4, 8446b7a7ea67486288d174eea7a3b3c6; no voters: 
I20240502 01:53:17.329035 14366 raft_consensus.cc:2781] T 01fb93c5dab3401fa5a103ead4dd9708 P 5359629f948641f4b56e35ad6b6a15c4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:17.329164 14366 raft_consensus.cc:482] T 01fb93c5dab3401fa5a103ead4dd9708 P 5359629f948641f4b56e35ad6b6a15c4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:17.329262 14366 raft_consensus.cc:3037] T 01fb93c5dab3401fa5a103ead4dd9708 P 5359629f948641f4b56e35ad6b6a15c4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:17.331310 14366 raft_consensus.cc:504] T 01fb93c5dab3401fa5a103ead4dd9708 P 5359629f948641f4b56e35ad6b6a15c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } }
I20240502 01:53:17.331858 14366 leader_election.cc:288] T 01fb93c5dab3401fa5a103ead4dd9708 P 5359629f948641f4b56e35ad6b6a15c4 [CANDIDATE]: Term 1 election: Requested vote from peers 8446b7a7ea67486288d174eea7a3b3c6 (127.9.176.194:34851)
I20240502 01:53:17.332453 13604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01fb93c5dab3401fa5a103ead4dd9708" candidate_uuid: "5359629f948641f4b56e35ad6b6a15c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8446b7a7ea67486288d174eea7a3b3c6"
I20240502 01:53:17.332648 13604 raft_consensus.cc:3037] T 01fb93c5dab3401fa5a103ead4dd9708 P 8446b7a7ea67486288d174eea7a3b3c6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:17.334491 13604 raft_consensus.cc:2454] T 01fb93c5dab3401fa5a103ead4dd9708 P 8446b7a7ea67486288d174eea7a3b3c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5359629f948641f4b56e35ad6b6a15c4 in term 1.
I20240502 01:53:17.334952 13402 leader_election.cc:302] T 01fb93c5dab3401fa5a103ead4dd9708 P 5359629f948641f4b56e35ad6b6a15c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 5359629f948641f4b56e35ad6b6a15c4, 8446b7a7ea67486288d174eea7a3b3c6; no voters: 
I20240502 01:53:17.335255 14366 raft_consensus.cc:2781] T 01fb93c5dab3401fa5a103ead4dd9708 P 5359629f948641f4b56e35ad6b6a15c4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:17.335733 14366 raft_consensus.cc:686] T 01fb93c5dab3401fa5a103ead4dd9708 P 5359629f948641f4b56e35ad6b6a15c4 [term 1 LEADER]: Becoming Leader. State: Replica: 5359629f948641f4b56e35ad6b6a15c4, State: Running, Role: LEADER
I20240502 01:53:17.336297 14366 consensus_queue.cc:237] T 01fb93c5dab3401fa5a103ead4dd9708 P 5359629f948641f4b56e35ad6b6a15c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } }
I20240502 01:53:17.340996 13326 catalog_manager.cc:5484] T 01fb93c5dab3401fa5a103ead4dd9708 P 5359629f948641f4b56e35ad6b6a15c4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5359629f948641f4b56e35ad6b6a15c4 (127.9.176.193). New cstate: current_term: 1 leader_uuid: "5359629f948641f4b56e35ad6b6a15c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } health_report { overall_health: UNKNOWN } } }
W20240502 01:53:17.422574 13783 tablet.cc:2369] T d4511dd1019c4bdcbf9fc7e4bfe2afb7 P b4abc88ed2e54830ae3a317629916117: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 01:53:17.429227 13651 tablet.cc:2369] T 01fb93c5dab3401fa5a103ead4dd9708 P 8446b7a7ea67486288d174eea7a3b3c6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:53:17.528270 14371 raft_consensus.cc:482] T d4511dd1019c4bdcbf9fc7e4bfe2afb7 P 8446b7a7ea67486288d174eea7a3b3c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:17.528501 14371 raft_consensus.cc:504] T d4511dd1019c4bdcbf9fc7e4bfe2afb7 P 8446b7a7ea67486288d174eea7a3b3c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } }
I20240502 01:53:17.530066 14371 leader_election.cc:288] T d4511dd1019c4bdcbf9fc7e4bfe2afb7 P 8446b7a7ea67486288d174eea7a3b3c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b4abc88ed2e54830ae3a317629916117 (127.9.176.195:41211)
I20240502 01:53:17.536898 13735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4511dd1019c4bdcbf9fc7e4bfe2afb7" candidate_uuid: "8446b7a7ea67486288d174eea7a3b3c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4abc88ed2e54830ae3a317629916117" is_pre_election: true
I20240502 01:53:17.537375 13735 raft_consensus.cc:2454] T d4511dd1019c4bdcbf9fc7e4bfe2afb7 P b4abc88ed2e54830ae3a317629916117 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8446b7a7ea67486288d174eea7a3b3c6 in term 0.
I20240502 01:53:17.538017 13538 leader_election.cc:302] T d4511dd1019c4bdcbf9fc7e4bfe2afb7 P 8446b7a7ea67486288d174eea7a3b3c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 8446b7a7ea67486288d174eea7a3b3c6, b4abc88ed2e54830ae3a317629916117; no voters: 
I20240502 01:53:17.538512 14371 raft_consensus.cc:2781] T d4511dd1019c4bdcbf9fc7e4bfe2afb7 P 8446b7a7ea67486288d174eea7a3b3c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:17.538645 14371 raft_consensus.cc:482] T d4511dd1019c4bdcbf9fc7e4bfe2afb7 P 8446b7a7ea67486288d174eea7a3b3c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:17.538730 14371 raft_consensus.cc:3037] T d4511dd1019c4bdcbf9fc7e4bfe2afb7 P 8446b7a7ea67486288d174eea7a3b3c6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:17.541040 14371 raft_consensus.cc:504] T d4511dd1019c4bdcbf9fc7e4bfe2afb7 P 8446b7a7ea67486288d174eea7a3b3c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } }
I20240502 01:53:17.541558 14371 leader_election.cc:288] T d4511dd1019c4bdcbf9fc7e4bfe2afb7 P 8446b7a7ea67486288d174eea7a3b3c6 [CANDIDATE]: Term 1 election: Requested vote from peers b4abc88ed2e54830ae3a317629916117 (127.9.176.195:41211)
I20240502 01:53:17.542114 13735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4511dd1019c4bdcbf9fc7e4bfe2afb7" candidate_uuid: "8446b7a7ea67486288d174eea7a3b3c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4abc88ed2e54830ae3a317629916117"
I20240502 01:53:17.542290 13735 raft_consensus.cc:3037] T d4511dd1019c4bdcbf9fc7e4bfe2afb7 P b4abc88ed2e54830ae3a317629916117 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:17.544370 13735 raft_consensus.cc:2454] T d4511dd1019c4bdcbf9fc7e4bfe2afb7 P b4abc88ed2e54830ae3a317629916117 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8446b7a7ea67486288d174eea7a3b3c6 in term 1.
I20240502 01:53:17.544826 13538 leader_election.cc:302] T d4511dd1019c4bdcbf9fc7e4bfe2afb7 P 8446b7a7ea67486288d174eea7a3b3c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 8446b7a7ea67486288d174eea7a3b3c6, b4abc88ed2e54830ae3a317629916117; no voters: 
I20240502 01:53:17.545163 14371 raft_consensus.cc:2781] T d4511dd1019c4bdcbf9fc7e4bfe2afb7 P 8446b7a7ea67486288d174eea7a3b3c6 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:17.545686 14371 raft_consensus.cc:686] T d4511dd1019c4bdcbf9fc7e4bfe2afb7 P 8446b7a7ea67486288d174eea7a3b3c6 [term 1 LEADER]: Becoming Leader. State: Replica: 8446b7a7ea67486288d174eea7a3b3c6, State: Running, Role: LEADER
I20240502 01:53:17.546197 14371 consensus_queue.cc:237] T d4511dd1019c4bdcbf9fc7e4bfe2afb7 P 8446b7a7ea67486288d174eea7a3b3c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } }
I20240502 01:53:17.550421 13326 catalog_manager.cc:5484] T d4511dd1019c4bdcbf9fc7e4bfe2afb7 P 8446b7a7ea67486288d174eea7a3b3c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8446b7a7ea67486288d174eea7a3b3c6 (127.9.176.194). New cstate: current_term: 1 leader_uuid: "8446b7a7ea67486288d174eea7a3b3c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:17.824731 14387 consensus_queue.cc:1035] T daed3123a96a496da83a6110e6943264 P b4abc88ed2e54830ae3a317629916117 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:17.921262 14392 consensus_queue.cc:1035] T 01fb93c5dab3401fa5a103ead4dd9708 P 5359629f948641f4b56e35ad6b6a15c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:53:17.985447 14386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:17.985829 14386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:17.993513 14386 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 01:53:17.997856 14371 consensus_queue.cc:1035] T d4511dd1019c4bdcbf9fc7e4bfe2afb7 P 8446b7a7ea67486288d174eea7a3b3c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_0'

INSERT report
    rows total: 2000
    time total: 204.668 ms
  time per row: 0.102334 ms
I20240502 01:53:18.355370 13321 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47160:
name: "rebalance_test_table_1"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 01:53:18.365475 13451 tablet_service.cc:1467] Processing CreateTablet for tablet edf0aa94bfb94d499015897713ed94fa (DEFAULT_TABLE table=rebalance_test_table_1 [id=8ebdfe08a0b244819813a774f9d721fc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:18.365561 13714 tablet_service.cc:1467] Processing CreateTablet for tablet edf0aa94bfb94d499015897713ed94fa (DEFAULT_TABLE table=rebalance_test_table_1 [id=8ebdfe08a0b244819813a774f9d721fc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:18.365989 13451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edf0aa94bfb94d499015897713ed94fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:18.365991 13714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edf0aa94bfb94d499015897713ed94fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:18.366413 13583 tablet_service.cc:1467] Processing CreateTablet for tablet 866d771c178a450e84189c64ddfca9d7 (DEFAULT_TABLE table=rebalance_test_table_1 [id=8ebdfe08a0b244819813a774f9d721fc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:18.366789 13583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 866d771c178a450e84189c64ddfca9d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:18.371215 13450 tablet_service.cc:1467] Processing CreateTablet for tablet b38a1fb4d1694c609a932e8816fc3514 (DEFAULT_TABLE table=rebalance_test_table_1 [id=8ebdfe08a0b244819813a774f9d721fc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:18.371929 13450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b38a1fb4d1694c609a932e8816fc3514. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:18.372483 13715 tablet_service.cc:1467] Processing CreateTablet for tablet b38a1fb4d1694c609a932e8816fc3514 (DEFAULT_TABLE table=rebalance_test_table_1 [id=8ebdfe08a0b244819813a774f9d721fc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:18.372902 13715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b38a1fb4d1694c609a932e8816fc3514. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:18.373102 13449 tablet_service.cc:1467] Processing CreateTablet for tablet 866d771c178a450e84189c64ddfca9d7 (DEFAULT_TABLE table=rebalance_test_table_1 [id=8ebdfe08a0b244819813a774f9d721fc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:18.373672 13449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 866d771c178a450e84189c64ddfca9d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:18.381695 14449 tablet_bootstrap.cc:492] T b38a1fb4d1694c609a932e8816fc3514 P b4abc88ed2e54830ae3a317629916117: Bootstrap starting.
I20240502 01:53:18.382762 14447 tablet_bootstrap.cc:492] T 866d771c178a450e84189c64ddfca9d7 P 8446b7a7ea67486288d174eea7a3b3c6: Bootstrap starting.
I20240502 01:53:18.383651 14448 tablet_bootstrap.cc:492] T edf0aa94bfb94d499015897713ed94fa P 5359629f948641f4b56e35ad6b6a15c4: Bootstrap starting.
I20240502 01:53:18.384801 14447 tablet_bootstrap.cc:654] T 866d771c178a450e84189c64ddfca9d7 P 8446b7a7ea67486288d174eea7a3b3c6: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:18.384953 14449 tablet_bootstrap.cc:654] T b38a1fb4d1694c609a932e8816fc3514 P b4abc88ed2e54830ae3a317629916117: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:18.385998 14448 tablet_bootstrap.cc:654] T edf0aa94bfb94d499015897713ed94fa P 5359629f948641f4b56e35ad6b6a15c4: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:18.388185 14449 tablet_bootstrap.cc:492] T b38a1fb4d1694c609a932e8816fc3514 P b4abc88ed2e54830ae3a317629916117: No bootstrap required, opened a new log
I20240502 01:53:18.388360 14449 ts_tablet_manager.cc:1367] T b38a1fb4d1694c609a932e8816fc3514 P b4abc88ed2e54830ae3a317629916117: Time spent bootstrapping tablet: real 0.007s	user 0.005s	sys 0.000s
I20240502 01:53:18.388502 14448 tablet_bootstrap.cc:492] T edf0aa94bfb94d499015897713ed94fa P 5359629f948641f4b56e35ad6b6a15c4: No bootstrap required, opened a new log
I20240502 01:53:18.388720 14448 ts_tablet_manager.cc:1367] T edf0aa94bfb94d499015897713ed94fa P 5359629f948641f4b56e35ad6b6a15c4: Time spent bootstrapping tablet: real 0.005s	user 0.000s	sys 0.004s
I20240502 01:53:18.389124 14447 tablet_bootstrap.cc:492] T 866d771c178a450e84189c64ddfca9d7 P 8446b7a7ea67486288d174eea7a3b3c6: No bootstrap required, opened a new log
I20240502 01:53:18.389286 14447 ts_tablet_manager.cc:1367] T 866d771c178a450e84189c64ddfca9d7 P 8446b7a7ea67486288d174eea7a3b3c6: Time spent bootstrapping tablet: real 0.007s	user 0.005s	sys 0.000s
I20240502 01:53:18.389225 14449 raft_consensus.cc:348] T b38a1fb4d1694c609a932e8816fc3514 P b4abc88ed2e54830ae3a317629916117 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } }
I20240502 01:53:18.389456 14449 raft_consensus.cc:374] T b38a1fb4d1694c609a932e8816fc3514 P b4abc88ed2e54830ae3a317629916117 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:18.389567 14449 raft_consensus.cc:729] T b38a1fb4d1694c609a932e8816fc3514 P b4abc88ed2e54830ae3a317629916117 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4abc88ed2e54830ae3a317629916117, State: Initialized, Role: FOLLOWER
I20240502 01:53:18.389480 14448 raft_consensus.cc:348] T edf0aa94bfb94d499015897713ed94fa P 5359629f948641f4b56e35ad6b6a15c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } }
I20240502 01:53:18.389679 14448 raft_consensus.cc:374] T edf0aa94bfb94d499015897713ed94fa P 5359629f948641f4b56e35ad6b6a15c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:18.389816 14448 raft_consensus.cc:729] T edf0aa94bfb94d499015897713ed94fa P 5359629f948641f4b56e35ad6b6a15c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5359629f948641f4b56e35ad6b6a15c4, State: Initialized, Role: FOLLOWER
I20240502 01:53:18.389823 14449 consensus_queue.cc:260] T b38a1fb4d1694c609a932e8816fc3514 P b4abc88ed2e54830ae3a317629916117 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } }
I20240502 01:53:18.390054 14448 consensus_queue.cc:260] T edf0aa94bfb94d499015897713ed94fa P 5359629f948641f4b56e35ad6b6a15c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } }
I20240502 01:53:18.390101 14447 raft_consensus.cc:348] T 866d771c178a450e84189c64ddfca9d7 P 8446b7a7ea67486288d174eea7a3b3c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } }
I20240502 01:53:18.390318 14447 raft_consensus.cc:374] T 866d771c178a450e84189c64ddfca9d7 P 8446b7a7ea67486288d174eea7a3b3c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:18.390420 14447 raft_consensus.cc:729] T 866d771c178a450e84189c64ddfca9d7 P 8446b7a7ea67486288d174eea7a3b3c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8446b7a7ea67486288d174eea7a3b3c6, State: Initialized, Role: FOLLOWER
I20240502 01:53:18.390643 14449 ts_tablet_manager.cc:1398] T b38a1fb4d1694c609a932e8816fc3514 P b4abc88ed2e54830ae3a317629916117: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 01:53:18.390637 14447 consensus_queue.cc:260] T 866d771c178a450e84189c64ddfca9d7 P 8446b7a7ea67486288d174eea7a3b3c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } }
I20240502 01:53:18.391256 14447 ts_tablet_manager.cc:1398] T 866d771c178a450e84189c64ddfca9d7 P 8446b7a7ea67486288d174eea7a3b3c6: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 01:53:18.391387 14448 ts_tablet_manager.cc:1398] T edf0aa94bfb94d499015897713ed94fa P 5359629f948641f4b56e35ad6b6a15c4: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.002s
I20240502 01:53:18.392374 14449 tablet_bootstrap.cc:492] T edf0aa94bfb94d499015897713ed94fa P b4abc88ed2e54830ae3a317629916117: Bootstrap starting.
I20240502 01:53:18.392778 14448 tablet_bootstrap.cc:492] T 866d771c178a450e84189c64ddfca9d7 P 5359629f948641f4b56e35ad6b6a15c4: Bootstrap starting.
I20240502 01:53:18.394508 14448 tablet_bootstrap.cc:654] T 866d771c178a450e84189c64ddfca9d7 P 5359629f948641f4b56e35ad6b6a15c4: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:18.395635 14449 tablet_bootstrap.cc:654] T edf0aa94bfb94d499015897713ed94fa P b4abc88ed2e54830ae3a317629916117: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:18.400771 14449 tablet_bootstrap.cc:492] T edf0aa94bfb94d499015897713ed94fa P b4abc88ed2e54830ae3a317629916117: No bootstrap required, opened a new log
I20240502 01:53:18.400903 14448 tablet_bootstrap.cc:492] T 866d771c178a450e84189c64ddfca9d7 P 5359629f948641f4b56e35ad6b6a15c4: No bootstrap required, opened a new log
I20240502 01:53:18.400935 14449 ts_tablet_manager.cc:1367] T edf0aa94bfb94d499015897713ed94fa P b4abc88ed2e54830ae3a317629916117: Time spent bootstrapping tablet: real 0.010s	user 0.003s	sys 0.003s
I20240502 01:53:18.401073 14448 ts_tablet_manager.cc:1367] T 866d771c178a450e84189c64ddfca9d7 P 5359629f948641f4b56e35ad6b6a15c4: Time spent bootstrapping tablet: real 0.009s	user 0.007s	sys 0.001s
I20240502 01:53:18.401648 14449 raft_consensus.cc:348] T edf0aa94bfb94d499015897713ed94fa P b4abc88ed2e54830ae3a317629916117 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } }
I20240502 01:53:18.401712 14448 raft_consensus.cc:348] T 866d771c178a450e84189c64ddfca9d7 P 5359629f948641f4b56e35ad6b6a15c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } }
I20240502 01:53:18.401831 14449 raft_consensus.cc:374] T edf0aa94bfb94d499015897713ed94fa P b4abc88ed2e54830ae3a317629916117 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:18.401901 14449 raft_consensus.cc:729] T edf0aa94bfb94d499015897713ed94fa P b4abc88ed2e54830ae3a317629916117 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4abc88ed2e54830ae3a317629916117, State: Initialized, Role: FOLLOWER
I20240502 01:53:18.401923 14448 raft_consensus.cc:374] T 866d771c178a450e84189c64ddfca9d7 P 5359629f948641f4b56e35ad6b6a15c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:18.401989 14448 raft_consensus.cc:729] T 866d771c178a450e84189c64ddfca9d7 P 5359629f948641f4b56e35ad6b6a15c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5359629f948641f4b56e35ad6b6a15c4, State: Initialized, Role: FOLLOWER
I20240502 01:53:18.402148 14449 consensus_queue.cc:260] T edf0aa94bfb94d499015897713ed94fa P b4abc88ed2e54830ae3a317629916117 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } }
I20240502 01:53:18.402240 14448 consensus_queue.cc:260] T 866d771c178a450e84189c64ddfca9d7 P 5359629f948641f4b56e35ad6b6a15c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } }
I20240502 01:53:18.402681 14449 ts_tablet_manager.cc:1398] T edf0aa94bfb94d499015897713ed94fa P b4abc88ed2e54830ae3a317629916117: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 01:53:18.402833 14448 ts_tablet_manager.cc:1398] T 866d771c178a450e84189c64ddfca9d7 P 5359629f948641f4b56e35ad6b6a15c4: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240502 01:53:18.403146 14448 tablet_bootstrap.cc:492] T b38a1fb4d1694c609a932e8816fc3514 P 5359629f948641f4b56e35ad6b6a15c4: Bootstrap starting.
I20240502 01:53:18.406252 14448 tablet_bootstrap.cc:654] T b38a1fb4d1694c609a932e8816fc3514 P 5359629f948641f4b56e35ad6b6a15c4: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:18.408402 14448 tablet_bootstrap.cc:492] T b38a1fb4d1694c609a932e8816fc3514 P 5359629f948641f4b56e35ad6b6a15c4: No bootstrap required, opened a new log
I20240502 01:53:18.408538 14448 ts_tablet_manager.cc:1367] T b38a1fb4d1694c609a932e8816fc3514 P 5359629f948641f4b56e35ad6b6a15c4: Time spent bootstrapping tablet: real 0.005s	user 0.002s	sys 0.002s
I20240502 01:53:18.409111 14448 raft_consensus.cc:348] T b38a1fb4d1694c609a932e8816fc3514 P 5359629f948641f4b56e35ad6b6a15c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } }
I20240502 01:53:18.409278 14448 raft_consensus.cc:374] T b38a1fb4d1694c609a932e8816fc3514 P 5359629f948641f4b56e35ad6b6a15c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:18.409370 14448 raft_consensus.cc:729] T b38a1fb4d1694c609a932e8816fc3514 P 5359629f948641f4b56e35ad6b6a15c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5359629f948641f4b56e35ad6b6a15c4, State: Initialized, Role: FOLLOWER
I20240502 01:53:18.409627 14448 consensus_queue.cc:260] T b38a1fb4d1694c609a932e8816fc3514 P 5359629f948641f4b56e35ad6b6a15c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } }
I20240502 01:53:18.410161 14448 ts_tablet_manager.cc:1398] T b38a1fb4d1694c609a932e8816fc3514 P 5359629f948641f4b56e35ad6b6a15c4: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 01:53:18.428133 14387 raft_consensus.cc:482] T edf0aa94bfb94d499015897713ed94fa P b4abc88ed2e54830ae3a317629916117 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:18.428375 14387 raft_consensus.cc:504] T edf0aa94bfb94d499015897713ed94fa P b4abc88ed2e54830ae3a317629916117 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } }
I20240502 01:53:18.429114 14387 leader_election.cc:288] T edf0aa94bfb94d499015897713ed94fa P b4abc88ed2e54830ae3a317629916117 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5359629f948641f4b56e35ad6b6a15c4 (127.9.176.193:33849)
I20240502 01:53:18.435592 13471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edf0aa94bfb94d499015897713ed94fa" candidate_uuid: "b4abc88ed2e54830ae3a317629916117" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5359629f948641f4b56e35ad6b6a15c4" is_pre_election: true
I20240502 01:53:18.435875 13471 raft_consensus.cc:2454] T edf0aa94bfb94d499015897713ed94fa P 5359629f948641f4b56e35ad6b6a15c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4abc88ed2e54830ae3a317629916117 in term 0.
I20240502 01:53:18.436352 13671 leader_election.cc:302] T edf0aa94bfb94d499015897713ed94fa P b4abc88ed2e54830ae3a317629916117 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 5359629f948641f4b56e35ad6b6a15c4, b4abc88ed2e54830ae3a317629916117; no voters: 
I20240502 01:53:18.436749 14387 raft_consensus.cc:2781] T edf0aa94bfb94d499015897713ed94fa P b4abc88ed2e54830ae3a317629916117 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:18.436879 14387 raft_consensus.cc:482] T edf0aa94bfb94d499015897713ed94fa P b4abc88ed2e54830ae3a317629916117 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:18.437012 14387 raft_consensus.cc:3037] T edf0aa94bfb94d499015897713ed94fa P b4abc88ed2e54830ae3a317629916117 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:18.439250 14387 raft_consensus.cc:504] T edf0aa94bfb94d499015897713ed94fa P b4abc88ed2e54830ae3a317629916117 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } }
I20240502 01:53:18.439815 14387 leader_election.cc:288] T edf0aa94bfb94d499015897713ed94fa P b4abc88ed2e54830ae3a317629916117 [CANDIDATE]: Term 1 election: Requested vote from peers 5359629f948641f4b56e35ad6b6a15c4 (127.9.176.193:33849)
I20240502 01:53:18.440408 13471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edf0aa94bfb94d499015897713ed94fa" candidate_uuid: "b4abc88ed2e54830ae3a317629916117" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5359629f948641f4b56e35ad6b6a15c4"
I20240502 01:53:18.440634 13471 raft_consensus.cc:3037] T edf0aa94bfb94d499015897713ed94fa P 5359629f948641f4b56e35ad6b6a15c4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:18.442586 13471 raft_consensus.cc:2454] T edf0aa94bfb94d499015897713ed94fa P 5359629f948641f4b56e35ad6b6a15c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4abc88ed2e54830ae3a317629916117 in term 1.
I20240502 01:53:18.443058 13671 leader_election.cc:302] T edf0aa94bfb94d499015897713ed94fa P b4abc88ed2e54830ae3a317629916117 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 5359629f948641f4b56e35ad6b6a15c4, b4abc88ed2e54830ae3a317629916117; no voters: 
I20240502 01:53:18.443377 14387 raft_consensus.cc:2781] T edf0aa94bfb94d499015897713ed94fa P b4abc88ed2e54830ae3a317629916117 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:18.443552 14387 raft_consensus.cc:686] T edf0aa94bfb94d499015897713ed94fa P b4abc88ed2e54830ae3a317629916117 [term 1 LEADER]: Becoming Leader. State: Replica: b4abc88ed2e54830ae3a317629916117, State: Running, Role: LEADER
I20240502 01:53:18.443830 14387 consensus_queue.cc:237] T edf0aa94bfb94d499015897713ed94fa P b4abc88ed2e54830ae3a317629916117 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } }
I20240502 01:53:18.446600 13321 catalog_manager.cc:5484] T edf0aa94bfb94d499015897713ed94fa P b4abc88ed2e54830ae3a317629916117 reported cstate change: term changed from 0 to 1, leader changed from <none> to b4abc88ed2e54830ae3a317629916117 (127.9.176.195). New cstate: current_term: 1 leader_uuid: "b4abc88ed2e54830ae3a317629916117" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } health_report { overall_health: HEALTHY } } }
I20240502 01:53:18.575585 14394 raft_consensus.cc:482] T 866d771c178a450e84189c64ddfca9d7 P 5359629f948641f4b56e35ad6b6a15c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:18.575837 14394 raft_consensus.cc:504] T 866d771c178a450e84189c64ddfca9d7 P 5359629f948641f4b56e35ad6b6a15c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } }
I20240502 01:53:18.576540 14394 leader_election.cc:288] T 866d771c178a450e84189c64ddfca9d7 P 5359629f948641f4b56e35ad6b6a15c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8446b7a7ea67486288d174eea7a3b3c6 (127.9.176.194:34851)
I20240502 01:53:18.577275 13603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "866d771c178a450e84189c64ddfca9d7" candidate_uuid: "5359629f948641f4b56e35ad6b6a15c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8446b7a7ea67486288d174eea7a3b3c6" is_pre_election: true
I20240502 01:53:18.577522 13603 raft_consensus.cc:2454] T 866d771c178a450e84189c64ddfca9d7 P 8446b7a7ea67486288d174eea7a3b3c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5359629f948641f4b56e35ad6b6a15c4 in term 0.
I20240502 01:53:18.578033 13402 leader_election.cc:302] T 866d771c178a450e84189c64ddfca9d7 P 5359629f948641f4b56e35ad6b6a15c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 5359629f948641f4b56e35ad6b6a15c4, 8446b7a7ea67486288d174eea7a3b3c6; no voters: 
I20240502 01:53:18.578361 14394 raft_consensus.cc:2781] T 866d771c178a450e84189c64ddfca9d7 P 5359629f948641f4b56e35ad6b6a15c4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:18.578487 14394 raft_consensus.cc:482] T 866d771c178a450e84189c64ddfca9d7 P 5359629f948641f4b56e35ad6b6a15c4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:18.578596 14394 raft_consensus.cc:3037] T 866d771c178a450e84189c64ddfca9d7 P 5359629f948641f4b56e35ad6b6a15c4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:18.580708 14394 raft_consensus.cc:504] T 866d771c178a450e84189c64ddfca9d7 P 5359629f948641f4b56e35ad6b6a15c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } }
I20240502 01:53:18.581204 14394 leader_election.cc:288] T 866d771c178a450e84189c64ddfca9d7 P 5359629f948641f4b56e35ad6b6a15c4 [CANDIDATE]: Term 1 election: Requested vote from peers 8446b7a7ea67486288d174eea7a3b3c6 (127.9.176.194:34851)
I20240502 01:53:18.581801 13603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "866d771c178a450e84189c64ddfca9d7" candidate_uuid: "5359629f948641f4b56e35ad6b6a15c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8446b7a7ea67486288d174eea7a3b3c6"
I20240502 01:53:18.582002 13603 raft_consensus.cc:3037] T 866d771c178a450e84189c64ddfca9d7 P 8446b7a7ea67486288d174eea7a3b3c6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:18.584076 13603 raft_consensus.cc:2454] T 866d771c178a450e84189c64ddfca9d7 P 8446b7a7ea67486288d174eea7a3b3c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5359629f948641f4b56e35ad6b6a15c4 in term 1.
I20240502 01:53:18.584539 13402 leader_election.cc:302] T 866d771c178a450e84189c64ddfca9d7 P 5359629f948641f4b56e35ad6b6a15c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 5359629f948641f4b56e35ad6b6a15c4, 8446b7a7ea67486288d174eea7a3b3c6; no voters: 
I20240502 01:53:18.584878 14394 raft_consensus.cc:2781] T 866d771c178a450e84189c64ddfca9d7 P 5359629f948641f4b56e35ad6b6a15c4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:18.585028 14394 raft_consensus.cc:686] T 866d771c178a450e84189c64ddfca9d7 P 5359629f948641f4b56e35ad6b6a15c4 [term 1 LEADER]: Becoming Leader. State: Replica: 5359629f948641f4b56e35ad6b6a15c4, State: Running, Role: LEADER
I20240502 01:53:18.585305 14394 consensus_queue.cc:237] T 866d771c178a450e84189c64ddfca9d7 P 5359629f948641f4b56e35ad6b6a15c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } }
I20240502 01:53:18.588277 13321 catalog_manager.cc:5484] T 866d771c178a450e84189c64ddfca9d7 P 5359629f948641f4b56e35ad6b6a15c4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5359629f948641f4b56e35ad6b6a15c4 (127.9.176.193). New cstate: current_term: 1 leader_uuid: "5359629f948641f4b56e35ad6b6a15c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:18.644697 14394 raft_consensus.cc:482] T b38a1fb4d1694c609a932e8816fc3514 P 5359629f948641f4b56e35ad6b6a15c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:18.644981 14394 raft_consensus.cc:504] T b38a1fb4d1694c609a932e8816fc3514 P 5359629f948641f4b56e35ad6b6a15c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } }
I20240502 01:53:18.645713 14394 leader_election.cc:288] T b38a1fb4d1694c609a932e8816fc3514 P 5359629f948641f4b56e35ad6b6a15c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b4abc88ed2e54830ae3a317629916117 (127.9.176.195:41211)
I20240502 01:53:18.652221 13735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b38a1fb4d1694c609a932e8816fc3514" candidate_uuid: "5359629f948641f4b56e35ad6b6a15c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4abc88ed2e54830ae3a317629916117" is_pre_election: true
I20240502 01:53:18.652508 13735 raft_consensus.cc:2454] T b38a1fb4d1694c609a932e8816fc3514 P b4abc88ed2e54830ae3a317629916117 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5359629f948641f4b56e35ad6b6a15c4 in term 0.
I20240502 01:53:18.653070 13404 leader_election.cc:302] T b38a1fb4d1694c609a932e8816fc3514 P 5359629f948641f4b56e35ad6b6a15c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 5359629f948641f4b56e35ad6b6a15c4, b4abc88ed2e54830ae3a317629916117; no voters: 
I20240502 01:53:18.653419 14394 raft_consensus.cc:2781] T b38a1fb4d1694c609a932e8816fc3514 P 5359629f948641f4b56e35ad6b6a15c4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:18.653517 14394 raft_consensus.cc:482] T b38a1fb4d1694c609a932e8816fc3514 P 5359629f948641f4b56e35ad6b6a15c4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:18.653595 14394 raft_consensus.cc:3037] T b38a1fb4d1694c609a932e8816fc3514 P 5359629f948641f4b56e35ad6b6a15c4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:18.655720 14394 raft_consensus.cc:504] T b38a1fb4d1694c609a932e8816fc3514 P 5359629f948641f4b56e35ad6b6a15c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } }
I20240502 01:53:18.656327 14394 leader_election.cc:288] T b38a1fb4d1694c609a932e8816fc3514 P 5359629f948641f4b56e35ad6b6a15c4 [CANDIDATE]: Term 1 election: Requested vote from peers b4abc88ed2e54830ae3a317629916117 (127.9.176.195:41211)
I20240502 01:53:18.657032 13735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b38a1fb4d1694c609a932e8816fc3514" candidate_uuid: "5359629f948641f4b56e35ad6b6a15c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4abc88ed2e54830ae3a317629916117"
I20240502 01:53:18.657253 13735 raft_consensus.cc:3037] T b38a1fb4d1694c609a932e8816fc3514 P b4abc88ed2e54830ae3a317629916117 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:18.659294 13735 raft_consensus.cc:2454] T b38a1fb4d1694c609a932e8816fc3514 P b4abc88ed2e54830ae3a317629916117 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5359629f948641f4b56e35ad6b6a15c4 in term 1.
I20240502 01:53:18.659888 13404 leader_election.cc:302] T b38a1fb4d1694c609a932e8816fc3514 P 5359629f948641f4b56e35ad6b6a15c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 5359629f948641f4b56e35ad6b6a15c4, b4abc88ed2e54830ae3a317629916117; no voters: 
I20240502 01:53:18.660266 14394 raft_consensus.cc:2781] T b38a1fb4d1694c609a932e8816fc3514 P 5359629f948641f4b56e35ad6b6a15c4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:18.660460 14394 raft_consensus.cc:686] T b38a1fb4d1694c609a932e8816fc3514 P 5359629f948641f4b56e35ad6b6a15c4 [term 1 LEADER]: Becoming Leader. State: Replica: 5359629f948641f4b56e35ad6b6a15c4, State: Running, Role: LEADER
I20240502 01:53:18.660751 14394 consensus_queue.cc:237] T b38a1fb4d1694c609a932e8816fc3514 P 5359629f948641f4b56e35ad6b6a15c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } }
I20240502 01:53:18.663558 13321 catalog_manager.cc:5484] T b38a1fb4d1694c609a932e8816fc3514 P 5359629f948641f4b56e35ad6b6a15c4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5359629f948641f4b56e35ad6b6a15c4 (127.9.176.193). New cstate: current_term: 1 leader_uuid: "5359629f948641f4b56e35ad6b6a15c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:18.822851 14387 consensus_queue.cc:1035] T edf0aa94bfb94d499015897713ed94fa P b4abc88ed2e54830ae3a317629916117 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:53:19.069185 14461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:19.069582 14461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:19.077682 14461 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240502 01:53:19.113960 14394 consensus_queue.cc:1035] T b38a1fb4d1694c609a932e8816fc3514 P 5359629f948641f4b56e35ad6b6a15c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:19.167136 14394 consensus_queue.cc:1035] T 866d771c178a450e84189c64ddfca9d7 P 5359629f948641f4b56e35ad6b6a15c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:19.255293 14489 raft_consensus.cc:482] T d38a05f7ffa94cc5abc50a76a83692b5 P 5359629f948641f4b56e35ad6b6a15c4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:19.255595 14489 raft_consensus.cc:504] T d38a05f7ffa94cc5abc50a76a83692b5 P 5359629f948641f4b56e35ad6b6a15c4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 41329 } }
I20240502 01:53:19.256546 14489 leader_election.cc:288] T d38a05f7ffa94cc5abc50a76a83692b5 P 5359629f948641f4b56e35ad6b6a15c4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ce90a4bd06ba41dd8e5aa227de9780a5 (127.9.176.196:41329)
W20240502 01:53:19.302886 13402 leader_election.cc:334] T d38a05f7ffa94cc5abc50a76a83692b5 P 5359629f948641f4b56e35ad6b6a15c4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ce90a4bd06ba41dd8e5aa227de9780a5 (127.9.176.196:41329): Network error: Client connection negotiation failed: client connection to 127.9.176.196:41329: connect: Connection refused (error 111)
I20240502 01:53:19.303211 13402 leader_election.cc:302] T d38a05f7ffa94cc5abc50a76a83692b5 P 5359629f948641f4b56e35ad6b6a15c4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 5359629f948641f4b56e35ad6b6a15c4; no voters: ce90a4bd06ba41dd8e5aa227de9780a5
I20240502 01:53:19.304454 14489 raft_consensus.cc:2726] T d38a05f7ffa94cc5abc50a76a83692b5 P 5359629f948641f4b56e35ad6b6a15c4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority

INSERT report
    rows total: 2000
    time total: 226.273 ms
  time per row: 0.113137 ms
I20240502 01:53:19.459875 13326 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47160:
name: "rebalance_test_table_2"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 01:53:19.470161 13583 tablet_service.cc:1467] Processing CreateTablet for tablet 57765e8e3ff7438ca9e588db8e685a03 (DEFAULT_TABLE table=rebalance_test_table_2 [id=c4a6875e444a446dbf9576f2a5d024c0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:19.470237 13715 tablet_service.cc:1467] Processing CreateTablet for tablet 57765e8e3ff7438ca9e588db8e685a03 (DEFAULT_TABLE table=rebalance_test_table_2 [id=c4a6875e444a446dbf9576f2a5d024c0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:19.470681 13715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57765e8e3ff7438ca9e588db8e685a03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:19.470681 13583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57765e8e3ff7438ca9e588db8e685a03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:19.473629 13714 tablet_service.cc:1467] Processing CreateTablet for tablet 8477215e8d4043c581c4ce7e5aa0fbd6 (DEFAULT_TABLE table=rebalance_test_table_2 [id=c4a6875e444a446dbf9576f2a5d024c0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:19.473992 13582 tablet_service.cc:1467] Processing CreateTablet for tablet 8477215e8d4043c581c4ce7e5aa0fbd6 (DEFAULT_TABLE table=rebalance_test_table_2 [id=c4a6875e444a446dbf9576f2a5d024c0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:19.474159 13714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8477215e8d4043c581c4ce7e5aa0fbd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:19.474429 13582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8477215e8d4043c581c4ce7e5aa0fbd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:19.481669 13584 tablet_service.cc:1467] Processing CreateTablet for tablet bdf6095f84aa497ba5db95d55c010085 (DEFAULT_TABLE table=rebalance_test_table_2 [id=c4a6875e444a446dbf9576f2a5d024c0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:19.481837 13451 tablet_service.cc:1467] Processing CreateTablet for tablet bdf6095f84aa497ba5db95d55c010085 (DEFAULT_TABLE table=rebalance_test_table_2 [id=c4a6875e444a446dbf9576f2a5d024c0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:19.482045 14514 tablet_bootstrap.cc:492] T 57765e8e3ff7438ca9e588db8e685a03 P 8446b7a7ea67486288d174eea7a3b3c6: Bootstrap starting.
I20240502 01:53:19.482226 13584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdf6095f84aa497ba5db95d55c010085. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:19.482244 13451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdf6095f84aa497ba5db95d55c010085. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:19.482785 14515 tablet_bootstrap.cc:492] T 8477215e8d4043c581c4ce7e5aa0fbd6 P b4abc88ed2e54830ae3a317629916117: Bootstrap starting.
I20240502 01:53:19.484582 14514 tablet_bootstrap.cc:654] T 57765e8e3ff7438ca9e588db8e685a03 P 8446b7a7ea67486288d174eea7a3b3c6: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:19.484763 14515 tablet_bootstrap.cc:654] T 8477215e8d4043c581c4ce7e5aa0fbd6 P b4abc88ed2e54830ae3a317629916117: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:19.494000 14518 tablet_bootstrap.cc:492] T bdf6095f84aa497ba5db95d55c010085 P 5359629f948641f4b56e35ad6b6a15c4: Bootstrap starting.
I20240502 01:53:19.496516 14518 tablet_bootstrap.cc:654] T bdf6095f84aa497ba5db95d55c010085 P 5359629f948641f4b56e35ad6b6a15c4: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:19.498556 14515 tablet_bootstrap.cc:492] T 8477215e8d4043c581c4ce7e5aa0fbd6 P b4abc88ed2e54830ae3a317629916117: No bootstrap required, opened a new log
I20240502 01:53:19.498721 14515 ts_tablet_manager.cc:1367] T 8477215e8d4043c581c4ce7e5aa0fbd6 P b4abc88ed2e54830ae3a317629916117: Time spent bootstrapping tablet: real 0.016s	user 0.007s	sys 0.008s
I20240502 01:53:19.499548 14514 tablet_bootstrap.cc:492] T 57765e8e3ff7438ca9e588db8e685a03 P 8446b7a7ea67486288d174eea7a3b3c6: No bootstrap required, opened a new log
I20240502 01:53:19.499811 14514 ts_tablet_manager.cc:1367] T 57765e8e3ff7438ca9e588db8e685a03 P 8446b7a7ea67486288d174eea7a3b3c6: Time spent bootstrapping tablet: real 0.018s	user 0.013s	sys 0.004s
I20240502 01:53:19.499734 14515 raft_consensus.cc:348] T 8477215e8d4043c581c4ce7e5aa0fbd6 P b4abc88ed2e54830ae3a317629916117 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } }
I20240502 01:53:19.500007 14515 raft_consensus.cc:374] T 8477215e8d4043c581c4ce7e5aa0fbd6 P b4abc88ed2e54830ae3a317629916117 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:19.500146 14515 raft_consensus.cc:729] T 8477215e8d4043c581c4ce7e5aa0fbd6 P b4abc88ed2e54830ae3a317629916117 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4abc88ed2e54830ae3a317629916117, State: Initialized, Role: FOLLOWER
I20240502 01:53:19.500470 14515 consensus_queue.cc:260] T 8477215e8d4043c581c4ce7e5aa0fbd6 P b4abc88ed2e54830ae3a317629916117 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } }
I20240502 01:53:19.500869 14514 raft_consensus.cc:348] T 57765e8e3ff7438ca9e588db8e685a03 P 8446b7a7ea67486288d174eea7a3b3c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } }
I20240502 01:53:19.501149 14514 raft_consensus.cc:374] T 57765e8e3ff7438ca9e588db8e685a03 P 8446b7a7ea67486288d174eea7a3b3c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:19.501174 14515 ts_tablet_manager.cc:1398] T 8477215e8d4043c581c4ce7e5aa0fbd6 P b4abc88ed2e54830ae3a317629916117: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 01:53:19.501302 14514 raft_consensus.cc:729] T 57765e8e3ff7438ca9e588db8e685a03 P 8446b7a7ea67486288d174eea7a3b3c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8446b7a7ea67486288d174eea7a3b3c6, State: Initialized, Role: FOLLOWER
I20240502 01:53:19.501672 14515 tablet_bootstrap.cc:492] T 57765e8e3ff7438ca9e588db8e685a03 P b4abc88ed2e54830ae3a317629916117: Bootstrap starting.
I20240502 01:53:19.501575 14514 consensus_queue.cc:260] T 57765e8e3ff7438ca9e588db8e685a03 P 8446b7a7ea67486288d174eea7a3b3c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } }
I20240502 01:53:19.502960 14514 ts_tablet_manager.cc:1398] T 57765e8e3ff7438ca9e588db8e685a03 P 8446b7a7ea67486288d174eea7a3b3c6: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 01:53:19.503325 14514 tablet_bootstrap.cc:492] T 8477215e8d4043c581c4ce7e5aa0fbd6 P 8446b7a7ea67486288d174eea7a3b3c6: Bootstrap starting.
I20240502 01:53:19.503767 14515 tablet_bootstrap.cc:654] T 57765e8e3ff7438ca9e588db8e685a03 P b4abc88ed2e54830ae3a317629916117: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:19.505440 14514 tablet_bootstrap.cc:654] T 8477215e8d4043c581c4ce7e5aa0fbd6 P 8446b7a7ea67486288d174eea7a3b3c6: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:19.507772 14518 tablet_bootstrap.cc:492] T bdf6095f84aa497ba5db95d55c010085 P 5359629f948641f4b56e35ad6b6a15c4: No bootstrap required, opened a new log
I20240502 01:53:19.508023 14518 ts_tablet_manager.cc:1367] T bdf6095f84aa497ba5db95d55c010085 P 5359629f948641f4b56e35ad6b6a15c4: Time spent bootstrapping tablet: real 0.014s	user 0.007s	sys 0.003s
I20240502 01:53:19.509330 14518 raft_consensus.cc:348] T bdf6095f84aa497ba5db95d55c010085 P 5359629f948641f4b56e35ad6b6a15c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } }
I20240502 01:53:19.509804 14518 raft_consensus.cc:374] T bdf6095f84aa497ba5db95d55c010085 P 5359629f948641f4b56e35ad6b6a15c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:19.509913 14518 raft_consensus.cc:729] T bdf6095f84aa497ba5db95d55c010085 P 5359629f948641f4b56e35ad6b6a15c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5359629f948641f4b56e35ad6b6a15c4, State: Initialized, Role: FOLLOWER
I20240502 01:53:19.510161 14518 consensus_queue.cc:260] T bdf6095f84aa497ba5db95d55c010085 P 5359629f948641f4b56e35ad6b6a15c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } }
I20240502 01:53:19.510928 14518 ts_tablet_manager.cc:1398] T bdf6095f84aa497ba5db95d55c010085 P 5359629f948641f4b56e35ad6b6a15c4: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 01:53:19.512583 14515 tablet_bootstrap.cc:492] T 57765e8e3ff7438ca9e588db8e685a03 P b4abc88ed2e54830ae3a317629916117: No bootstrap required, opened a new log
I20240502 01:53:19.512866 14515 ts_tablet_manager.cc:1367] T 57765e8e3ff7438ca9e588db8e685a03 P b4abc88ed2e54830ae3a317629916117: Time spent bootstrapping tablet: real 0.011s	user 0.002s	sys 0.009s
I20240502 01:53:19.513562 14514 tablet_bootstrap.cc:492] T 8477215e8d4043c581c4ce7e5aa0fbd6 P 8446b7a7ea67486288d174eea7a3b3c6: No bootstrap required, opened a new log
I20240502 01:53:19.513768 14514 ts_tablet_manager.cc:1367] T 8477215e8d4043c581c4ce7e5aa0fbd6 P 8446b7a7ea67486288d174eea7a3b3c6: Time spent bootstrapping tablet: real 0.010s	user 0.004s	sys 0.004s
I20240502 01:53:19.513865 14515 raft_consensus.cc:348] T 57765e8e3ff7438ca9e588db8e685a03 P b4abc88ed2e54830ae3a317629916117 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } }
I20240502 01:53:19.514130 14515 raft_consensus.cc:374] T 57765e8e3ff7438ca9e588db8e685a03 P b4abc88ed2e54830ae3a317629916117 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:19.514219 14515 raft_consensus.cc:729] T 57765e8e3ff7438ca9e588db8e685a03 P b4abc88ed2e54830ae3a317629916117 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4abc88ed2e54830ae3a317629916117, State: Initialized, Role: FOLLOWER
I20240502 01:53:19.514503 14515 consensus_queue.cc:260] T 57765e8e3ff7438ca9e588db8e685a03 P b4abc88ed2e54830ae3a317629916117 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } }
I20240502 01:53:19.514719 14514 raft_consensus.cc:348] T 8477215e8d4043c581c4ce7e5aa0fbd6 P 8446b7a7ea67486288d174eea7a3b3c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } }
I20240502 01:53:19.515005 14514 raft_consensus.cc:374] T 8477215e8d4043c581c4ce7e5aa0fbd6 P 8446b7a7ea67486288d174eea7a3b3c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:19.515120 14514 raft_consensus.cc:729] T 8477215e8d4043c581c4ce7e5aa0fbd6 P 8446b7a7ea67486288d174eea7a3b3c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8446b7a7ea67486288d174eea7a3b3c6, State: Initialized, Role: FOLLOWER
I20240502 01:53:19.515178 14515 ts_tablet_manager.cc:1398] T 57765e8e3ff7438ca9e588db8e685a03 P b4abc88ed2e54830ae3a317629916117: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 01:53:19.515412 14514 consensus_queue.cc:260] T 8477215e8d4043c581c4ce7e5aa0fbd6 P 8446b7a7ea67486288d174eea7a3b3c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } }
I20240502 01:53:19.516041 14514 ts_tablet_manager.cc:1398] T 8477215e8d4043c581c4ce7e5aa0fbd6 P 8446b7a7ea67486288d174eea7a3b3c6: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.003s
I20240502 01:53:19.516379 14514 tablet_bootstrap.cc:492] T bdf6095f84aa497ba5db95d55c010085 P 8446b7a7ea67486288d174eea7a3b3c6: Bootstrap starting.
I20240502 01:53:19.518429 14514 tablet_bootstrap.cc:654] T bdf6095f84aa497ba5db95d55c010085 P 8446b7a7ea67486288d174eea7a3b3c6: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:19.521299 14514 tablet_bootstrap.cc:492] T bdf6095f84aa497ba5db95d55c010085 P 8446b7a7ea67486288d174eea7a3b3c6: No bootstrap required, opened a new log
I20240502 01:53:19.521449 14514 ts_tablet_manager.cc:1367] T bdf6095f84aa497ba5db95d55c010085 P 8446b7a7ea67486288d174eea7a3b3c6: Time spent bootstrapping tablet: real 0.005s	user 0.000s	sys 0.004s
I20240502 01:53:19.522156 14514 raft_consensus.cc:348] T bdf6095f84aa497ba5db95d55c010085 P 8446b7a7ea67486288d174eea7a3b3c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } }
I20240502 01:53:19.522326 14514 raft_consensus.cc:374] T bdf6095f84aa497ba5db95d55c010085 P 8446b7a7ea67486288d174eea7a3b3c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:19.522439 14514 raft_consensus.cc:729] T bdf6095f84aa497ba5db95d55c010085 P 8446b7a7ea67486288d174eea7a3b3c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8446b7a7ea67486288d174eea7a3b3c6, State: Initialized, Role: FOLLOWER
I20240502 01:53:19.522678 14514 consensus_queue.cc:260] T bdf6095f84aa497ba5db95d55c010085 P 8446b7a7ea67486288d174eea7a3b3c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } }
I20240502 01:53:19.523209 14514 ts_tablet_manager.cc:1398] T bdf6095f84aa497ba5db95d55c010085 P 8446b7a7ea67486288d174eea7a3b3c6: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 01:53:19.660531 14384 raft_consensus.cc:482] T 8477215e8d4043c581c4ce7e5aa0fbd6 P 8446b7a7ea67486288d174eea7a3b3c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:19.660758 14384 raft_consensus.cc:504] T 8477215e8d4043c581c4ce7e5aa0fbd6 P 8446b7a7ea67486288d174eea7a3b3c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } }
I20240502 01:53:19.661386 14384 leader_election.cc:288] T 8477215e8d4043c581c4ce7e5aa0fbd6 P 8446b7a7ea67486288d174eea7a3b3c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b4abc88ed2e54830ae3a317629916117 (127.9.176.195:41211)
I20240502 01:53:19.662078 13735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8477215e8d4043c581c4ce7e5aa0fbd6" candidate_uuid: "8446b7a7ea67486288d174eea7a3b3c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4abc88ed2e54830ae3a317629916117" is_pre_election: true
I20240502 01:53:19.662312 13735 raft_consensus.cc:2454] T 8477215e8d4043c581c4ce7e5aa0fbd6 P b4abc88ed2e54830ae3a317629916117 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8446b7a7ea67486288d174eea7a3b3c6 in term 0.
I20240502 01:53:19.662770 13538 leader_election.cc:302] T 8477215e8d4043c581c4ce7e5aa0fbd6 P 8446b7a7ea67486288d174eea7a3b3c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 8446b7a7ea67486288d174eea7a3b3c6, b4abc88ed2e54830ae3a317629916117; no voters: 
I20240502 01:53:19.663118 14384 raft_consensus.cc:2781] T 8477215e8d4043c581c4ce7e5aa0fbd6 P 8446b7a7ea67486288d174eea7a3b3c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:19.663247 14384 raft_consensus.cc:482] T 8477215e8d4043c581c4ce7e5aa0fbd6 P 8446b7a7ea67486288d174eea7a3b3c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:19.663378 14384 raft_consensus.cc:3037] T 8477215e8d4043c581c4ce7e5aa0fbd6 P 8446b7a7ea67486288d174eea7a3b3c6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:19.666503 14384 raft_consensus.cc:504] T 8477215e8d4043c581c4ce7e5aa0fbd6 P 8446b7a7ea67486288d174eea7a3b3c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } }
I20240502 01:53:19.667127 14384 leader_election.cc:288] T 8477215e8d4043c581c4ce7e5aa0fbd6 P 8446b7a7ea67486288d174eea7a3b3c6 [CANDIDATE]: Term 1 election: Requested vote from peers b4abc88ed2e54830ae3a317629916117 (127.9.176.195:41211)
I20240502 01:53:19.667726 13735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8477215e8d4043c581c4ce7e5aa0fbd6" candidate_uuid: "8446b7a7ea67486288d174eea7a3b3c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4abc88ed2e54830ae3a317629916117"
I20240502 01:53:19.667969 13735 raft_consensus.cc:3037] T 8477215e8d4043c581c4ce7e5aa0fbd6 P b4abc88ed2e54830ae3a317629916117 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:19.670032 13735 raft_consensus.cc:2454] T 8477215e8d4043c581c4ce7e5aa0fbd6 P b4abc88ed2e54830ae3a317629916117 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8446b7a7ea67486288d174eea7a3b3c6 in term 1.
I20240502 01:53:19.670511 13538 leader_election.cc:302] T 8477215e8d4043c581c4ce7e5aa0fbd6 P 8446b7a7ea67486288d174eea7a3b3c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 8446b7a7ea67486288d174eea7a3b3c6, b4abc88ed2e54830ae3a317629916117; no voters: 
I20240502 01:53:19.670869 14384 raft_consensus.cc:2781] T 8477215e8d4043c581c4ce7e5aa0fbd6 P 8446b7a7ea67486288d174eea7a3b3c6 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:19.671336 14384 raft_consensus.cc:686] T 8477215e8d4043c581c4ce7e5aa0fbd6 P 8446b7a7ea67486288d174eea7a3b3c6 [term 1 LEADER]: Becoming Leader. State: Replica: 8446b7a7ea67486288d174eea7a3b3c6, State: Running, Role: LEADER
I20240502 01:53:19.671756 14384 consensus_queue.cc:237] T 8477215e8d4043c581c4ce7e5aa0fbd6 P 8446b7a7ea67486288d174eea7a3b3c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } }
I20240502 01:53:19.674822 13326 catalog_manager.cc:5484] T 8477215e8d4043c581c4ce7e5aa0fbd6 P 8446b7a7ea67486288d174eea7a3b3c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8446b7a7ea67486288d174eea7a3b3c6 (127.9.176.194). New cstate: current_term: 1 leader_uuid: "8446b7a7ea67486288d174eea7a3b3c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:19.865903 14387 raft_consensus.cc:482] T 57765e8e3ff7438ca9e588db8e685a03 P b4abc88ed2e54830ae3a317629916117 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:19.866102 14387 raft_consensus.cc:504] T 57765e8e3ff7438ca9e588db8e685a03 P b4abc88ed2e54830ae3a317629916117 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } }
I20240502 01:53:19.866590 14387 leader_election.cc:288] T 57765e8e3ff7438ca9e588db8e685a03 P b4abc88ed2e54830ae3a317629916117 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8446b7a7ea67486288d174eea7a3b3c6 (127.9.176.194:34851)
I20240502 01:53:19.867214 13604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57765e8e3ff7438ca9e588db8e685a03" candidate_uuid: "b4abc88ed2e54830ae3a317629916117" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8446b7a7ea67486288d174eea7a3b3c6" is_pre_election: true
I20240502 01:53:19.867439 13604 raft_consensus.cc:2454] T 57765e8e3ff7438ca9e588db8e685a03 P 8446b7a7ea67486288d174eea7a3b3c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4abc88ed2e54830ae3a317629916117 in term 0.
I20240502 01:53:19.867949 13668 leader_election.cc:302] T 57765e8e3ff7438ca9e588db8e685a03 P b4abc88ed2e54830ae3a317629916117 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 8446b7a7ea67486288d174eea7a3b3c6, b4abc88ed2e54830ae3a317629916117; no voters: 
I20240502 01:53:19.868265 14387 raft_consensus.cc:2781] T 57765e8e3ff7438ca9e588db8e685a03 P b4abc88ed2e54830ae3a317629916117 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:19.868355 14387 raft_consensus.cc:482] T 57765e8e3ff7438ca9e588db8e685a03 P b4abc88ed2e54830ae3a317629916117 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:19.868460 14387 raft_consensus.cc:3037] T 57765e8e3ff7438ca9e588db8e685a03 P b4abc88ed2e54830ae3a317629916117 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:19.870214 14387 raft_consensus.cc:504] T 57765e8e3ff7438ca9e588db8e685a03 P b4abc88ed2e54830ae3a317629916117 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } }
I20240502 01:53:19.870645 14387 leader_election.cc:288] T 57765e8e3ff7438ca9e588db8e685a03 P b4abc88ed2e54830ae3a317629916117 [CANDIDATE]: Term 1 election: Requested vote from peers 8446b7a7ea67486288d174eea7a3b3c6 (127.9.176.194:34851)
I20240502 01:53:19.871146 13604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57765e8e3ff7438ca9e588db8e685a03" candidate_uuid: "b4abc88ed2e54830ae3a317629916117" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8446b7a7ea67486288d174eea7a3b3c6"
I20240502 01:53:19.871335 13604 raft_consensus.cc:3037] T 57765e8e3ff7438ca9e588db8e685a03 P 8446b7a7ea67486288d174eea7a3b3c6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:19.873219 13604 raft_consensus.cc:2454] T 57765e8e3ff7438ca9e588db8e685a03 P 8446b7a7ea67486288d174eea7a3b3c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4abc88ed2e54830ae3a317629916117 in term 1.
I20240502 01:53:19.873612 13668 leader_election.cc:302] T 57765e8e3ff7438ca9e588db8e685a03 P b4abc88ed2e54830ae3a317629916117 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 8446b7a7ea67486288d174eea7a3b3c6, b4abc88ed2e54830ae3a317629916117; no voters: 
I20240502 01:53:19.873876 14387 raft_consensus.cc:2781] T 57765e8e3ff7438ca9e588db8e685a03 P b4abc88ed2e54830ae3a317629916117 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:19.874235 14387 raft_consensus.cc:686] T 57765e8e3ff7438ca9e588db8e685a03 P b4abc88ed2e54830ae3a317629916117 [term 1 LEADER]: Becoming Leader. State: Replica: b4abc88ed2e54830ae3a317629916117, State: Running, Role: LEADER
I20240502 01:53:19.874476 14387 consensus_queue.cc:237] T 57765e8e3ff7438ca9e588db8e685a03 P b4abc88ed2e54830ae3a317629916117 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } }
I20240502 01:53:19.876737 13326 catalog_manager.cc:5484] T 57765e8e3ff7438ca9e588db8e685a03 P b4abc88ed2e54830ae3a317629916117 reported cstate change: term changed from 0 to 1, leader changed from <none> to b4abc88ed2e54830ae3a317629916117 (127.9.176.195). New cstate: current_term: 1 leader_uuid: "b4abc88ed2e54830ae3a317629916117" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } health_report { overall_health: HEALTHY } } }
I20240502 01:53:19.903851 14384 raft_consensus.cc:482] T bdf6095f84aa497ba5db95d55c010085 P 8446b7a7ea67486288d174eea7a3b3c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:19.904079 14384 raft_consensus.cc:504] T bdf6095f84aa497ba5db95d55c010085 P 8446b7a7ea67486288d174eea7a3b3c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } }
I20240502 01:53:19.904568 14489 raft_consensus.cc:482] T bdf6095f84aa497ba5db95d55c010085 P 5359629f948641f4b56e35ad6b6a15c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:19.904656 14384 leader_election.cc:288] T bdf6095f84aa497ba5db95d55c010085 P 8446b7a7ea67486288d174eea7a3b3c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5359629f948641f4b56e35ad6b6a15c4 (127.9.176.193:33849)
I20240502 01:53:19.904719 14489 raft_consensus.cc:504] T bdf6095f84aa497ba5db95d55c010085 P 5359629f948641f4b56e35ad6b6a15c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } }
I20240502 01:53:19.905280 14489 leader_election.cc:288] T bdf6095f84aa497ba5db95d55c010085 P 5359629f948641f4b56e35ad6b6a15c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8446b7a7ea67486288d174eea7a3b3c6 (127.9.176.194:34851)
I20240502 01:53:19.905797 13604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdf6095f84aa497ba5db95d55c010085" candidate_uuid: "5359629f948641f4b56e35ad6b6a15c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8446b7a7ea67486288d174eea7a3b3c6" is_pre_election: true
I20240502 01:53:19.906076 13604 raft_consensus.cc:2454] T bdf6095f84aa497ba5db95d55c010085 P 8446b7a7ea67486288d174eea7a3b3c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5359629f948641f4b56e35ad6b6a15c4 in term 0.
I20240502 01:53:19.906499 13402 leader_election.cc:302] T bdf6095f84aa497ba5db95d55c010085 P 5359629f948641f4b56e35ad6b6a15c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 5359629f948641f4b56e35ad6b6a15c4, 8446b7a7ea67486288d174eea7a3b3c6; no voters: 
I20240502 01:53:19.906749 14489 raft_consensus.cc:2781] T bdf6095f84aa497ba5db95d55c010085 P 5359629f948641f4b56e35ad6b6a15c4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:19.906886 14489 raft_consensus.cc:482] T bdf6095f84aa497ba5db95d55c010085 P 5359629f948641f4b56e35ad6b6a15c4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:19.906981 14489 raft_consensus.cc:3037] T bdf6095f84aa497ba5db95d55c010085 P 5359629f948641f4b56e35ad6b6a15c4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:19.909456 14489 raft_consensus.cc:504] T bdf6095f84aa497ba5db95d55c010085 P 5359629f948641f4b56e35ad6b6a15c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } }
I20240502 01:53:19.909981 14489 leader_election.cc:288] T bdf6095f84aa497ba5db95d55c010085 P 5359629f948641f4b56e35ad6b6a15c4 [CANDIDATE]: Term 1 election: Requested vote from peers 8446b7a7ea67486288d174eea7a3b3c6 (127.9.176.194:34851)
I20240502 01:53:19.910578 13604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdf6095f84aa497ba5db95d55c010085" candidate_uuid: "5359629f948641f4b56e35ad6b6a15c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8446b7a7ea67486288d174eea7a3b3c6"
I20240502 01:53:19.910820 13604 raft_consensus.cc:3037] T bdf6095f84aa497ba5db95d55c010085 P 8446b7a7ea67486288d174eea7a3b3c6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:19.911748 13471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdf6095f84aa497ba5db95d55c010085" candidate_uuid: "8446b7a7ea67486288d174eea7a3b3c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5359629f948641f4b56e35ad6b6a15c4" is_pre_election: true
I20240502 01:53:19.912034 13471 raft_consensus.cc:2379] T bdf6095f84aa497ba5db95d55c010085 P 5359629f948641f4b56e35ad6b6a15c4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8446b7a7ea67486288d174eea7a3b3c6 in current term 1: Already voted for candidate 5359629f948641f4b56e35ad6b6a15c4 in this term.
I20240502 01:53:19.912528 13539 leader_election.cc:302] T bdf6095f84aa497ba5db95d55c010085 P 8446b7a7ea67486288d174eea7a3b3c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 8446b7a7ea67486288d174eea7a3b3c6; no voters: 5359629f948641f4b56e35ad6b6a15c4
I20240502 01:53:19.913518 13604 raft_consensus.cc:2454] T bdf6095f84aa497ba5db95d55c010085 P 8446b7a7ea67486288d174eea7a3b3c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5359629f948641f4b56e35ad6b6a15c4 in term 1.
I20240502 01:53:19.913796 14384 raft_consensus.cc:2726] T bdf6095f84aa497ba5db95d55c010085 P 8446b7a7ea67486288d174eea7a3b3c6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:53:19.914036 13402 leader_election.cc:302] T bdf6095f84aa497ba5db95d55c010085 P 5359629f948641f4b56e35ad6b6a15c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 5359629f948641f4b56e35ad6b6a15c4, 8446b7a7ea67486288d174eea7a3b3c6; no voters: 
I20240502 01:53:19.914412 14489 raft_consensus.cc:2781] T bdf6095f84aa497ba5db95d55c010085 P 5359629f948641f4b56e35ad6b6a15c4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:19.914746 14489 raft_consensus.cc:686] T bdf6095f84aa497ba5db95d55c010085 P 5359629f948641f4b56e35ad6b6a15c4 [term 1 LEADER]: Becoming Leader. State: Replica: 5359629f948641f4b56e35ad6b6a15c4, State: Running, Role: LEADER
I20240502 01:53:19.914983 14489 consensus_queue.cc:237] T bdf6095f84aa497ba5db95d55c010085 P 5359629f948641f4b56e35ad6b6a15c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } }
I20240502 01:53:19.918061 13326 catalog_manager.cc:5484] T bdf6095f84aa497ba5db95d55c010085 P 5359629f948641f4b56e35ad6b6a15c4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5359629f948641f4b56e35ad6b6a15c4 (127.9.176.193). New cstate: current_term: 1 leader_uuid: "5359629f948641f4b56e35ad6b6a15c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:20.194300 14384 consensus_queue.cc:1035] T 8477215e8d4043c581c4ce7e5aa0fbd6 P 8446b7a7ea67486288d174eea7a3b3c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:53:20.281252 14532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:20.281616 14532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:20.289378 14532 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240502 01:53:20.423962 14489 consensus_queue.cc:1035] T bdf6095f84aa497ba5db95d55c010085 P 5359629f948641f4b56e35ad6b6a15c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:20.426934 13602 raft_consensus.cc:1261] T 57765e8e3ff7438ca9e588db8e685a03 P 8446b7a7ea67486288d174eea7a3b3c6 [term 1 FOLLOWER]: Refusing update from remote peer b4abc88ed2e54830ae3a317629916117: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:53:20.427912 14387 consensus_queue.cc:1035] T 57765e8e3ff7438ca9e588db8e685a03 P b4abc88ed2e54830ae3a317629916117 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 262.474 ms
  time per row: 0.131237 ms
I20240502 01:53:20.704651 13320 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47160:
name: "rebalance_test_table_3"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 01:53:20.716099 13584 tablet_service.cc:1467] Processing CreateTablet for tablet 6682ed5cba8448c7b3cf74ab726a18e7 (DEFAULT_TABLE table=rebalance_test_table_3 [id=c824fe62874b40458b3bf6ce96e3242b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:20.716279 13450 tablet_service.cc:1467] Processing CreateTablet for tablet 4c0ab7c2995d4ccfa170f365b4e09b1d (DEFAULT_TABLE table=rebalance_test_table_3 [id=c824fe62874b40458b3bf6ce96e3242b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:20.716642 13584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6682ed5cba8448c7b3cf74ab726a18e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:20.716714 13450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c0ab7c2995d4ccfa170f365b4e09b1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:20.716785 13451 tablet_service.cc:1467] Processing CreateTablet for tablet eb1cf83294e044b49e115323de67ed42 (DEFAULT_TABLE table=rebalance_test_table_3 [id=c824fe62874b40458b3bf6ce96e3242b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:20.717489 13714 tablet_service.cc:1467] Processing CreateTablet for tablet eb1cf83294e044b49e115323de67ed42 (DEFAULT_TABLE table=rebalance_test_table_3 [id=c824fe62874b40458b3bf6ce96e3242b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:20.717903 13714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb1cf83294e044b49e115323de67ed42. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:20.719976 13715 tablet_service.cc:1467] Processing CreateTablet for tablet 4c0ab7c2995d4ccfa170f365b4e09b1d (DEFAULT_TABLE table=rebalance_test_table_3 [id=c824fe62874b40458b3bf6ce96e3242b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:20.720394 13715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c0ab7c2995d4ccfa170f365b4e09b1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:20.723470 13449 tablet_service.cc:1467] Processing CreateTablet for tablet 6682ed5cba8448c7b3cf74ab726a18e7 (DEFAULT_TABLE table=rebalance_test_table_3 [id=c824fe62874b40458b3bf6ce96e3242b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:20.723937 13449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6682ed5cba8448c7b3cf74ab726a18e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:20.724191 13451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb1cf83294e044b49e115323de67ed42. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:20.731298 14594 tablet_bootstrap.cc:492] T 4c0ab7c2995d4ccfa170f365b4e09b1d P 5359629f948641f4b56e35ad6b6a15c4: Bootstrap starting.
I20240502 01:53:20.739465 14594 tablet_bootstrap.cc:654] T 4c0ab7c2995d4ccfa170f365b4e09b1d P 5359629f948641f4b56e35ad6b6a15c4: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:20.744465 14595 tablet_bootstrap.cc:492] T 4c0ab7c2995d4ccfa170f365b4e09b1d P b4abc88ed2e54830ae3a317629916117: Bootstrap starting.
I20240502 01:53:20.747869 14595 tablet_bootstrap.cc:654] T 4c0ab7c2995d4ccfa170f365b4e09b1d P b4abc88ed2e54830ae3a317629916117: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:20.748628 14593 tablet_bootstrap.cc:492] T 6682ed5cba8448c7b3cf74ab726a18e7 P 8446b7a7ea67486288d174eea7a3b3c6: Bootstrap starting.
I20240502 01:53:20.750274 14593 tablet_bootstrap.cc:654] T 6682ed5cba8448c7b3cf74ab726a18e7 P 8446b7a7ea67486288d174eea7a3b3c6: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:20.753182 14593 tablet_bootstrap.cc:492] T 6682ed5cba8448c7b3cf74ab726a18e7 P 8446b7a7ea67486288d174eea7a3b3c6: No bootstrap required, opened a new log
I20240502 01:53:20.753453 14593 ts_tablet_manager.cc:1367] T 6682ed5cba8448c7b3cf74ab726a18e7 P 8446b7a7ea67486288d174eea7a3b3c6: Time spent bootstrapping tablet: real 0.005s	user 0.003s	sys 0.000s
I20240502 01:53:20.754408 14593 raft_consensus.cc:348] T 6682ed5cba8448c7b3cf74ab726a18e7 P 8446b7a7ea67486288d174eea7a3b3c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } } peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } }
I20240502 01:53:20.754719 14593 raft_consensus.cc:374] T 6682ed5cba8448c7b3cf74ab726a18e7 P 8446b7a7ea67486288d174eea7a3b3c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:20.754884 14593 raft_consensus.cc:729] T 6682ed5cba8448c7b3cf74ab726a18e7 P 8446b7a7ea67486288d174eea7a3b3c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8446b7a7ea67486288d174eea7a3b3c6, State: Initialized, Role: FOLLOWER
I20240502 01:53:20.755056 14594 tablet_bootstrap.cc:492] T 4c0ab7c2995d4ccfa170f365b4e09b1d P 5359629f948641f4b56e35ad6b6a15c4: No bootstrap required, opened a new log
I20240502 01:53:20.755260 14594 ts_tablet_manager.cc:1367] T 4c0ab7c2995d4ccfa170f365b4e09b1d P 5359629f948641f4b56e35ad6b6a15c4: Time spent bootstrapping tablet: real 0.024s	user 0.003s	sys 0.006s
I20240502 01:53:20.756292 14595 tablet_bootstrap.cc:492] T 4c0ab7c2995d4ccfa170f365b4e09b1d P b4abc88ed2e54830ae3a317629916117: No bootstrap required, opened a new log
I20240502 01:53:20.756376 14594 raft_consensus.cc:348] T 4c0ab7c2995d4ccfa170f365b4e09b1d P 5359629f948641f4b56e35ad6b6a15c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } }
I20240502 01:53:20.756533 14595 ts_tablet_manager.cc:1367] T 4c0ab7c2995d4ccfa170f365b4e09b1d P b4abc88ed2e54830ae3a317629916117: Time spent bootstrapping tablet: real 0.012s	user 0.005s	sys 0.000s
I20240502 01:53:20.756659 14594 raft_consensus.cc:374] T 4c0ab7c2995d4ccfa170f365b4e09b1d P 5359629f948641f4b56e35ad6b6a15c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:20.756814 14594 raft_consensus.cc:729] T 4c0ab7c2995d4ccfa170f365b4e09b1d P 5359629f948641f4b56e35ad6b6a15c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5359629f948641f4b56e35ad6b6a15c4, State: Initialized, Role: FOLLOWER
I20240502 01:53:20.757113 14594 consensus_queue.cc:260] T 4c0ab7c2995d4ccfa170f365b4e09b1d P 5359629f948641f4b56e35ad6b6a15c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } }
I20240502 01:53:20.757788 14594 ts_tablet_manager.cc:1398] T 4c0ab7c2995d4ccfa170f365b4e09b1d P 5359629f948641f4b56e35ad6b6a15c4: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240502 01:53:20.757886 14593 consensus_queue.cc:260] T 6682ed5cba8448c7b3cf74ab726a18e7 P 8446b7a7ea67486288d174eea7a3b3c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } } peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } }
I20240502 01:53:20.758204 14594 tablet_bootstrap.cc:492] T 6682ed5cba8448c7b3cf74ab726a18e7 P 5359629f948641f4b56e35ad6b6a15c4: Bootstrap starting.
I20240502 01:53:20.758579 14593 ts_tablet_manager.cc:1398] T 6682ed5cba8448c7b3cf74ab726a18e7 P 8446b7a7ea67486288d174eea7a3b3c6: Time spent starting tablet: real 0.005s	user 0.002s	sys 0.000s
I20240502 01:53:20.759970 14594 tablet_bootstrap.cc:654] T 6682ed5cba8448c7b3cf74ab726a18e7 P 5359629f948641f4b56e35ad6b6a15c4: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:20.764441 14595 raft_consensus.cc:348] T 4c0ab7c2995d4ccfa170f365b4e09b1d P b4abc88ed2e54830ae3a317629916117 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } }
I20240502 01:53:20.764802 14595 raft_consensus.cc:374] T 4c0ab7c2995d4ccfa170f365b4e09b1d P b4abc88ed2e54830ae3a317629916117 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:20.764932 14595 raft_consensus.cc:729] T 4c0ab7c2995d4ccfa170f365b4e09b1d P b4abc88ed2e54830ae3a317629916117 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4abc88ed2e54830ae3a317629916117, State: Initialized, Role: FOLLOWER
I20240502 01:53:20.765313 14595 consensus_queue.cc:260] T 4c0ab7c2995d4ccfa170f365b4e09b1d P b4abc88ed2e54830ae3a317629916117 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } }
I20240502 01:53:20.766220 14595 ts_tablet_manager.cc:1398] T 4c0ab7c2995d4ccfa170f365b4e09b1d P b4abc88ed2e54830ae3a317629916117: Time spent starting tablet: real 0.009s	user 0.002s	sys 0.000s
I20240502 01:53:20.766613 14595 tablet_bootstrap.cc:492] T eb1cf83294e044b49e115323de67ed42 P b4abc88ed2e54830ae3a317629916117: Bootstrap starting.
I20240502 01:53:20.768920 14595 tablet_bootstrap.cc:654] T eb1cf83294e044b49e115323de67ed42 P b4abc88ed2e54830ae3a317629916117: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:20.774432 14594 tablet_bootstrap.cc:492] T 6682ed5cba8448c7b3cf74ab726a18e7 P 5359629f948641f4b56e35ad6b6a15c4: No bootstrap required, opened a new log
I20240502 01:53:20.774652 14594 ts_tablet_manager.cc:1367] T 6682ed5cba8448c7b3cf74ab726a18e7 P 5359629f948641f4b56e35ad6b6a15c4: Time spent bootstrapping tablet: real 0.016s	user 0.006s	sys 0.009s
I20240502 01:53:20.775521 14595 tablet_bootstrap.cc:492] T eb1cf83294e044b49e115323de67ed42 P b4abc88ed2e54830ae3a317629916117: No bootstrap required, opened a new log
I20240502 01:53:20.775440 14594 raft_consensus.cc:348] T 6682ed5cba8448c7b3cf74ab726a18e7 P 5359629f948641f4b56e35ad6b6a15c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } } peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } }
I20240502 01:53:20.775751 14594 raft_consensus.cc:374] T 6682ed5cba8448c7b3cf74ab726a18e7 P 5359629f948641f4b56e35ad6b6a15c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:20.775867 14594 raft_consensus.cc:729] T 6682ed5cba8448c7b3cf74ab726a18e7 P 5359629f948641f4b56e35ad6b6a15c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5359629f948641f4b56e35ad6b6a15c4, State: Initialized, Role: FOLLOWER
I20240502 01:53:20.775887 14595 ts_tablet_manager.cc:1367] T eb1cf83294e044b49e115323de67ed42 P b4abc88ed2e54830ae3a317629916117: Time spent bootstrapping tablet: real 0.009s	user 0.001s	sys 0.003s
I20240502 01:53:20.776157 14594 consensus_queue.cc:260] T 6682ed5cba8448c7b3cf74ab726a18e7 P 5359629f948641f4b56e35ad6b6a15c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } } peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } }
I20240502 01:53:20.776743 14595 raft_consensus.cc:348] T eb1cf83294e044b49e115323de67ed42 P b4abc88ed2e54830ae3a317629916117 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } }
I20240502 01:53:20.776899 14594 ts_tablet_manager.cc:1398] T 6682ed5cba8448c7b3cf74ab726a18e7 P 5359629f948641f4b56e35ad6b6a15c4: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240502 01:53:20.777034 14595 raft_consensus.cc:374] T eb1cf83294e044b49e115323de67ed42 P b4abc88ed2e54830ae3a317629916117 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:20.777197 14595 raft_consensus.cc:729] T eb1cf83294e044b49e115323de67ed42 P b4abc88ed2e54830ae3a317629916117 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4abc88ed2e54830ae3a317629916117, State: Initialized, Role: FOLLOWER
I20240502 01:53:20.777271 14594 tablet_bootstrap.cc:492] T eb1cf83294e044b49e115323de67ed42 P 5359629f948641f4b56e35ad6b6a15c4: Bootstrap starting.
I20240502 01:53:20.777549 14595 consensus_queue.cc:260] T eb1cf83294e044b49e115323de67ed42 P b4abc88ed2e54830ae3a317629916117 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } }
I20240502 01:53:20.778347 14595 ts_tablet_manager.cc:1398] T eb1cf83294e044b49e115323de67ed42 P b4abc88ed2e54830ae3a317629916117: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 01:53:20.778834 14594 tablet_bootstrap.cc:654] T eb1cf83294e044b49e115323de67ed42 P 5359629f948641f4b56e35ad6b6a15c4: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:20.782202 14594 tablet_bootstrap.cc:492] T eb1cf83294e044b49e115323de67ed42 P 5359629f948641f4b56e35ad6b6a15c4: No bootstrap required, opened a new log
I20240502 01:53:20.782356 14594 ts_tablet_manager.cc:1367] T eb1cf83294e044b49e115323de67ed42 P 5359629f948641f4b56e35ad6b6a15c4: Time spent bootstrapping tablet: real 0.005s	user 0.003s	sys 0.001s
I20240502 01:53:20.783138 14594 raft_consensus.cc:348] T eb1cf83294e044b49e115323de67ed42 P 5359629f948641f4b56e35ad6b6a15c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } }
I20240502 01:53:20.783339 14594 raft_consensus.cc:374] T eb1cf83294e044b49e115323de67ed42 P 5359629f948641f4b56e35ad6b6a15c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:20.783423 14594 raft_consensus.cc:729] T eb1cf83294e044b49e115323de67ed42 P 5359629f948641f4b56e35ad6b6a15c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5359629f948641f4b56e35ad6b6a15c4, State: Initialized, Role: FOLLOWER
I20240502 01:53:20.783687 14594 consensus_queue.cc:260] T eb1cf83294e044b49e115323de67ed42 P 5359629f948641f4b56e35ad6b6a15c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } }
I20240502 01:53:20.784210 14594 ts_tablet_manager.cc:1398] T eb1cf83294e044b49e115323de67ed42 P 5359629f948641f4b56e35ad6b6a15c4: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 01:53:20.806591 14489 raft_consensus.cc:482] T 6682ed5cba8448c7b3cf74ab726a18e7 P 5359629f948641f4b56e35ad6b6a15c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:20.806802 14489 raft_consensus.cc:504] T 6682ed5cba8448c7b3cf74ab726a18e7 P 5359629f948641f4b56e35ad6b6a15c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } } peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } }
I20240502 01:53:20.807394 14489 leader_election.cc:288] T 6682ed5cba8448c7b3cf74ab726a18e7 P 5359629f948641f4b56e35ad6b6a15c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8446b7a7ea67486288d174eea7a3b3c6 (127.9.176.194:34851)
I20240502 01:53:20.808130 13604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6682ed5cba8448c7b3cf74ab726a18e7" candidate_uuid: "5359629f948641f4b56e35ad6b6a15c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8446b7a7ea67486288d174eea7a3b3c6" is_pre_election: true
I20240502 01:53:20.808377 13604 raft_consensus.cc:2454] T 6682ed5cba8448c7b3cf74ab726a18e7 P 8446b7a7ea67486288d174eea7a3b3c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5359629f948641f4b56e35ad6b6a15c4 in term 0.
I20240502 01:53:20.808799 13402 leader_election.cc:302] T 6682ed5cba8448c7b3cf74ab726a18e7 P 5359629f948641f4b56e35ad6b6a15c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 5359629f948641f4b56e35ad6b6a15c4, 8446b7a7ea67486288d174eea7a3b3c6; no voters: 
I20240502 01:53:20.809104 14489 raft_consensus.cc:2781] T 6682ed5cba8448c7b3cf74ab726a18e7 P 5359629f948641f4b56e35ad6b6a15c4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:20.809239 14489 raft_consensus.cc:482] T 6682ed5cba8448c7b3cf74ab726a18e7 P 5359629f948641f4b56e35ad6b6a15c4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:20.809327 14489 raft_consensus.cc:3037] T 6682ed5cba8448c7b3cf74ab726a18e7 P 5359629f948641f4b56e35ad6b6a15c4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:20.811422 14489 raft_consensus.cc:504] T 6682ed5cba8448c7b3cf74ab726a18e7 P 5359629f948641f4b56e35ad6b6a15c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } } peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } }
I20240502 01:53:20.811954 14489 leader_election.cc:288] T 6682ed5cba8448c7b3cf74ab726a18e7 P 5359629f948641f4b56e35ad6b6a15c4 [CANDIDATE]: Term 1 election: Requested vote from peers 8446b7a7ea67486288d174eea7a3b3c6 (127.9.176.194:34851)
I20240502 01:53:20.812577 13604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6682ed5cba8448c7b3cf74ab726a18e7" candidate_uuid: "5359629f948641f4b56e35ad6b6a15c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8446b7a7ea67486288d174eea7a3b3c6"
I20240502 01:53:20.812772 13604 raft_consensus.cc:3037] T 6682ed5cba8448c7b3cf74ab726a18e7 P 8446b7a7ea67486288d174eea7a3b3c6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:20.814869 13604 raft_consensus.cc:2454] T 6682ed5cba8448c7b3cf74ab726a18e7 P 8446b7a7ea67486288d174eea7a3b3c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5359629f948641f4b56e35ad6b6a15c4 in term 1.
I20240502 01:53:20.815307 13402 leader_election.cc:302] T 6682ed5cba8448c7b3cf74ab726a18e7 P 5359629f948641f4b56e35ad6b6a15c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 5359629f948641f4b56e35ad6b6a15c4, 8446b7a7ea67486288d174eea7a3b3c6; no voters: 
I20240502 01:53:20.815672 14489 raft_consensus.cc:2781] T 6682ed5cba8448c7b3cf74ab726a18e7 P 5359629f948641f4b56e35ad6b6a15c4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:20.815842 14489 raft_consensus.cc:686] T 6682ed5cba8448c7b3cf74ab726a18e7 P 5359629f948641f4b56e35ad6b6a15c4 [term 1 LEADER]: Becoming Leader. State: Replica: 5359629f948641f4b56e35ad6b6a15c4, State: Running, Role: LEADER
I20240502 01:53:20.816136 14489 consensus_queue.cc:237] T 6682ed5cba8448c7b3cf74ab726a18e7 P 5359629f948641f4b56e35ad6b6a15c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } } peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } }
I20240502 01:53:20.818696 13320 catalog_manager.cc:5484] T 6682ed5cba8448c7b3cf74ab726a18e7 P 5359629f948641f4b56e35ad6b6a15c4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5359629f948641f4b56e35ad6b6a15c4 (127.9.176.193). New cstate: current_term: 1 leader_uuid: "5359629f948641f4b56e35ad6b6a15c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } health_report { overall_health: HEALTHY } } }
I20240502 01:53:20.963552 14387 raft_consensus.cc:482] T 4c0ab7c2995d4ccfa170f365b4e09b1d P b4abc88ed2e54830ae3a317629916117 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:20.963835 14387 raft_consensus.cc:504] T 4c0ab7c2995d4ccfa170f365b4e09b1d P b4abc88ed2e54830ae3a317629916117 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } }
I20240502 01:53:20.964361 14387 leader_election.cc:288] T 4c0ab7c2995d4ccfa170f365b4e09b1d P b4abc88ed2e54830ae3a317629916117 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5359629f948641f4b56e35ad6b6a15c4 (127.9.176.193:33849)
I20240502 01:53:20.965070 13471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c0ab7c2995d4ccfa170f365b4e09b1d" candidate_uuid: "b4abc88ed2e54830ae3a317629916117" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5359629f948641f4b56e35ad6b6a15c4" is_pre_election: true
I20240502 01:53:20.965273 13471 raft_consensus.cc:2454] T 4c0ab7c2995d4ccfa170f365b4e09b1d P 5359629f948641f4b56e35ad6b6a15c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4abc88ed2e54830ae3a317629916117 in term 0.
I20240502 01:53:20.965694 13671 leader_election.cc:302] T 4c0ab7c2995d4ccfa170f365b4e09b1d P b4abc88ed2e54830ae3a317629916117 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 5359629f948641f4b56e35ad6b6a15c4, b4abc88ed2e54830ae3a317629916117; no voters: 
I20240502 01:53:20.966012 14387 raft_consensus.cc:2781] T 4c0ab7c2995d4ccfa170f365b4e09b1d P b4abc88ed2e54830ae3a317629916117 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:20.966111 14387 raft_consensus.cc:482] T 4c0ab7c2995d4ccfa170f365b4e09b1d P b4abc88ed2e54830ae3a317629916117 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:20.966223 14387 raft_consensus.cc:3037] T 4c0ab7c2995d4ccfa170f365b4e09b1d P b4abc88ed2e54830ae3a317629916117 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:20.968204 14387 raft_consensus.cc:504] T 4c0ab7c2995d4ccfa170f365b4e09b1d P b4abc88ed2e54830ae3a317629916117 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } }
I20240502 01:53:20.968770 14387 leader_election.cc:288] T 4c0ab7c2995d4ccfa170f365b4e09b1d P b4abc88ed2e54830ae3a317629916117 [CANDIDATE]: Term 1 election: Requested vote from peers 5359629f948641f4b56e35ad6b6a15c4 (127.9.176.193:33849)
I20240502 01:53:20.969377 13471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c0ab7c2995d4ccfa170f365b4e09b1d" candidate_uuid: "b4abc88ed2e54830ae3a317629916117" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5359629f948641f4b56e35ad6b6a15c4"
I20240502 01:53:20.969576 13471 raft_consensus.cc:3037] T 4c0ab7c2995d4ccfa170f365b4e09b1d P 5359629f948641f4b56e35ad6b6a15c4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:20.971530 13471 raft_consensus.cc:2454] T 4c0ab7c2995d4ccfa170f365b4e09b1d P 5359629f948641f4b56e35ad6b6a15c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4abc88ed2e54830ae3a317629916117 in term 1.
I20240502 01:53:20.972169 13671 leader_election.cc:302] T 4c0ab7c2995d4ccfa170f365b4e09b1d P b4abc88ed2e54830ae3a317629916117 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 5359629f948641f4b56e35ad6b6a15c4, b4abc88ed2e54830ae3a317629916117; no voters: 
I20240502 01:53:20.972646 14387 raft_consensus.cc:2781] T 4c0ab7c2995d4ccfa170f365b4e09b1d P b4abc88ed2e54830ae3a317629916117 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:20.972855 14387 raft_consensus.cc:686] T 4c0ab7c2995d4ccfa170f365b4e09b1d P b4abc88ed2e54830ae3a317629916117 [term 1 LEADER]: Becoming Leader. State: Replica: b4abc88ed2e54830ae3a317629916117, State: Running, Role: LEADER
I20240502 01:53:20.973152 14387 consensus_queue.cc:237] T 4c0ab7c2995d4ccfa170f365b4e09b1d P b4abc88ed2e54830ae3a317629916117 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } }
I20240502 01:53:20.975914 13320 catalog_manager.cc:5484] T 4c0ab7c2995d4ccfa170f365b4e09b1d P b4abc88ed2e54830ae3a317629916117 reported cstate change: term changed from 0 to 1, leader changed from <none> to b4abc88ed2e54830ae3a317629916117 (127.9.176.195). New cstate: current_term: 1 leader_uuid: "b4abc88ed2e54830ae3a317629916117" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } health_report { overall_health: HEALTHY } } }
I20240502 01:53:21.056849 14489 raft_consensus.cc:482] T eb1cf83294e044b49e115323de67ed42 P 5359629f948641f4b56e35ad6b6a15c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:21.057121 14489 raft_consensus.cc:504] T eb1cf83294e044b49e115323de67ed42 P 5359629f948641f4b56e35ad6b6a15c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } }
I20240502 01:53:21.057678 14489 leader_election.cc:288] T eb1cf83294e044b49e115323de67ed42 P 5359629f948641f4b56e35ad6b6a15c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b4abc88ed2e54830ae3a317629916117 (127.9.176.195:41211)
I20240502 01:53:21.058351 13735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb1cf83294e044b49e115323de67ed42" candidate_uuid: "5359629f948641f4b56e35ad6b6a15c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4abc88ed2e54830ae3a317629916117" is_pre_election: true
I20240502 01:53:21.058629 13735 raft_consensus.cc:2454] T eb1cf83294e044b49e115323de67ed42 P b4abc88ed2e54830ae3a317629916117 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5359629f948641f4b56e35ad6b6a15c4 in term 0.
I20240502 01:53:21.059103 13404 leader_election.cc:302] T eb1cf83294e044b49e115323de67ed42 P 5359629f948641f4b56e35ad6b6a15c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 5359629f948641f4b56e35ad6b6a15c4, b4abc88ed2e54830ae3a317629916117; no voters: 
I20240502 01:53:21.059430 14489 raft_consensus.cc:2781] T eb1cf83294e044b49e115323de67ed42 P 5359629f948641f4b56e35ad6b6a15c4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:21.059517 14489 raft_consensus.cc:482] T eb1cf83294e044b49e115323de67ed42 P 5359629f948641f4b56e35ad6b6a15c4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:21.059643 14489 raft_consensus.cc:3037] T eb1cf83294e044b49e115323de67ed42 P 5359629f948641f4b56e35ad6b6a15c4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:21.061702 14489 raft_consensus.cc:504] T eb1cf83294e044b49e115323de67ed42 P 5359629f948641f4b56e35ad6b6a15c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } }
I20240502 01:53:21.062191 14489 leader_election.cc:288] T eb1cf83294e044b49e115323de67ed42 P 5359629f948641f4b56e35ad6b6a15c4 [CANDIDATE]: Term 1 election: Requested vote from peers b4abc88ed2e54830ae3a317629916117 (127.9.176.195:41211)
I20240502 01:53:21.062774 13735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb1cf83294e044b49e115323de67ed42" candidate_uuid: "5359629f948641f4b56e35ad6b6a15c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4abc88ed2e54830ae3a317629916117"
I20240502 01:53:21.062978 13735 raft_consensus.cc:3037] T eb1cf83294e044b49e115323de67ed42 P b4abc88ed2e54830ae3a317629916117 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:21.065026 13735 raft_consensus.cc:2454] T eb1cf83294e044b49e115323de67ed42 P b4abc88ed2e54830ae3a317629916117 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5359629f948641f4b56e35ad6b6a15c4 in term 1.
I20240502 01:53:21.065464 13404 leader_election.cc:302] T eb1cf83294e044b49e115323de67ed42 P 5359629f948641f4b56e35ad6b6a15c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 5359629f948641f4b56e35ad6b6a15c4, b4abc88ed2e54830ae3a317629916117; no voters: 
I20240502 01:53:21.065742 14489 raft_consensus.cc:2781] T eb1cf83294e044b49e115323de67ed42 P 5359629f948641f4b56e35ad6b6a15c4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:21.065896 14489 raft_consensus.cc:686] T eb1cf83294e044b49e115323de67ed42 P 5359629f948641f4b56e35ad6b6a15c4 [term 1 LEADER]: Becoming Leader. State: Replica: 5359629f948641f4b56e35ad6b6a15c4, State: Running, Role: LEADER
I20240502 01:53:21.066145 14489 consensus_queue.cc:237] T eb1cf83294e044b49e115323de67ed42 P 5359629f948641f4b56e35ad6b6a15c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } }
I20240502 01:53:21.068650 13320 catalog_manager.cc:5484] T eb1cf83294e044b49e115323de67ed42 P 5359629f948641f4b56e35ad6b6a15c4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5359629f948641f4b56e35ad6b6a15c4 (127.9.176.193). New cstate: current_term: 1 leader_uuid: "5359629f948641f4b56e35ad6b6a15c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:21.226652 14489 consensus_queue.cc:1035] T 6682ed5cba8448c7b3cf74ab726a18e7 P 5359629f948641f4b56e35ad6b6a15c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:53:21.456573 14605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:21.456969 14605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:21.465457 14605 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240502 01:53:21.505461 14387 consensus_queue.cc:1035] T 4c0ab7c2995d4ccfa170f365b4e09b1d P b4abc88ed2e54830ae3a317629916117 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:21.573415 14489 consensus_queue.cc:1035] T eb1cf83294e044b49e115323de67ed42 P 5359629f948641f4b56e35ad6b6a15c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 199.27 ms
  time per row: 0.0996349 ms
I20240502 01:53:21.819022 13322 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47160:
name: "rebalance_test_table_4"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 01:53:21.828344 13584 tablet_service.cc:1467] Processing CreateTablet for tablet 930645182ab748d7b864fece1e858b3f (DEFAULT_TABLE table=rebalance_test_table_4 [id=7c97435ee22d42aaa3452b6e2abdaf75]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:21.828670 13715 tablet_service.cc:1467] Processing CreateTablet for tablet 930645182ab748d7b864fece1e858b3f (DEFAULT_TABLE table=rebalance_test_table_4 [id=7c97435ee22d42aaa3452b6e2abdaf75]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:21.828823 13584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 930645182ab748d7b864fece1e858b3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:21.829133 13715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 930645182ab748d7b864fece1e858b3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:21.829260 13451 tablet_service.cc:1467] Processing CreateTablet for tablet 71b0f9fbee82464994252126b2d631bc (DEFAULT_TABLE table=rebalance_test_table_4 [id=7c97435ee22d42aaa3452b6e2abdaf75]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:21.829674 13451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71b0f9fbee82464994252126b2d631bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:21.829902 13714 tablet_service.cc:1467] Processing CreateTablet for tablet f2af8f5d5e784e1988a957666844db6b (DEFAULT_TABLE table=rebalance_test_table_4 [id=7c97435ee22d42aaa3452b6e2abdaf75]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:21.830284 13714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2af8f5d5e784e1988a957666844db6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:21.832664 13583 tablet_service.cc:1467] Processing CreateTablet for tablet f2af8f5d5e784e1988a957666844db6b (DEFAULT_TABLE table=rebalance_test_table_4 [id=7c97435ee22d42aaa3452b6e2abdaf75]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:21.833067 13583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2af8f5d5e784e1988a957666844db6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:21.837803 13582 tablet_service.cc:1467] Processing CreateTablet for tablet 71b0f9fbee82464994252126b2d631bc (DEFAULT_TABLE table=rebalance_test_table_4 [id=7c97435ee22d42aaa3452b6e2abdaf75]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:21.838428 13582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71b0f9fbee82464994252126b2d631bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:21.844494 14489 raft_consensus.cc:482] T d38a05f7ffa94cc5abc50a76a83692b5 P 5359629f948641f4b56e35ad6b6a15c4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:21.844666 14489 raft_consensus.cc:504] T d38a05f7ffa94cc5abc50a76a83692b5 P 5359629f948641f4b56e35ad6b6a15c4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 41329 } }
I20240502 01:53:21.844866 14664 tablet_bootstrap.cc:492] T 930645182ab748d7b864fece1e858b3f P b4abc88ed2e54830ae3a317629916117: Bootstrap starting.
I20240502 01:53:21.845222 14489 leader_election.cc:288] T d38a05f7ffa94cc5abc50a76a83692b5 P 5359629f948641f4b56e35ad6b6a15c4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ce90a4bd06ba41dd8e5aa227de9780a5 (127.9.176.196:41329)
I20240502 01:53:21.847205 14664 tablet_bootstrap.cc:654] T 930645182ab748d7b864fece1e858b3f P b4abc88ed2e54830ae3a317629916117: Neither blocks nor log segments found. Creating new log.
W20240502 01:53:21.848105 13402 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.176.196:41329: connect: Connection refused (error 111) [suppressed 2 similar messages]
I20240502 01:53:21.848723 14665 tablet_bootstrap.cc:492] T f2af8f5d5e784e1988a957666844db6b P 8446b7a7ea67486288d174eea7a3b3c6: Bootstrap starting.
I20240502 01:53:21.851835 14665 tablet_bootstrap.cc:654] T f2af8f5d5e784e1988a957666844db6b P 8446b7a7ea67486288d174eea7a3b3c6: Neither blocks nor log segments found. Creating new log.
W20240502 01:53:21.852587 13402 leader_election.cc:334] T d38a05f7ffa94cc5abc50a76a83692b5 P 5359629f948641f4b56e35ad6b6a15c4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ce90a4bd06ba41dd8e5aa227de9780a5 (127.9.176.196:41329): Network error: Client connection negotiation failed: client connection to 127.9.176.196:41329: connect: Connection refused (error 111)
I20240502 01:53:21.852859 14663 tablet_bootstrap.cc:492] T 71b0f9fbee82464994252126b2d631bc P 5359629f948641f4b56e35ad6b6a15c4: Bootstrap starting.
I20240502 01:53:21.852824 13402 leader_election.cc:302] T d38a05f7ffa94cc5abc50a76a83692b5 P 5359629f948641f4b56e35ad6b6a15c4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 5359629f948641f4b56e35ad6b6a15c4; no voters: ce90a4bd06ba41dd8e5aa227de9780a5
I20240502 01:53:21.853328 14489 raft_consensus.cc:2726] T d38a05f7ffa94cc5abc50a76a83692b5 P 5359629f948641f4b56e35ad6b6a15c4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:53:21.854054 14664 tablet_bootstrap.cc:492] T 930645182ab748d7b864fece1e858b3f P b4abc88ed2e54830ae3a317629916117: No bootstrap required, opened a new log
I20240502 01:53:21.854301 14664 ts_tablet_manager.cc:1367] T 930645182ab748d7b864fece1e858b3f P b4abc88ed2e54830ae3a317629916117: Time spent bootstrapping tablet: real 0.010s	user 0.005s	sys 0.000s
I20240502 01:53:21.855098 14663 tablet_bootstrap.cc:654] T 71b0f9fbee82464994252126b2d631bc P 5359629f948641f4b56e35ad6b6a15c4: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:21.855005 14664 raft_consensus.cc:348] T 930645182ab748d7b864fece1e858b3f P b4abc88ed2e54830ae3a317629916117 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } }
I20240502 01:53:21.855158 14665 tablet_bootstrap.cc:492] T f2af8f5d5e784e1988a957666844db6b P 8446b7a7ea67486288d174eea7a3b3c6: No bootstrap required, opened a new log
I20240502 01:53:21.855195 14664 raft_consensus.cc:374] T 930645182ab748d7b864fece1e858b3f P b4abc88ed2e54830ae3a317629916117 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:21.855326 14664 raft_consensus.cc:729] T 930645182ab748d7b864fece1e858b3f P b4abc88ed2e54830ae3a317629916117 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4abc88ed2e54830ae3a317629916117, State: Initialized, Role: FOLLOWER
I20240502 01:53:21.855331 14665 ts_tablet_manager.cc:1367] T f2af8f5d5e784e1988a957666844db6b P 8446b7a7ea67486288d174eea7a3b3c6: Time spent bootstrapping tablet: real 0.007s	user 0.000s	sys 0.004s
I20240502 01:53:21.855659 14664 consensus_queue.cc:260] T 930645182ab748d7b864fece1e858b3f P b4abc88ed2e54830ae3a317629916117 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } }
I20240502 01:53:21.856308 14665 raft_consensus.cc:348] T f2af8f5d5e784e1988a957666844db6b P 8446b7a7ea67486288d174eea7a3b3c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } }
I20240502 01:53:21.856463 14664 ts_tablet_manager.cc:1398] T 930645182ab748d7b864fece1e858b3f P b4abc88ed2e54830ae3a317629916117: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 01:53:21.856608 14665 raft_consensus.cc:374] T f2af8f5d5e784e1988a957666844db6b P 8446b7a7ea67486288d174eea7a3b3c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:21.856765 14664 tablet_bootstrap.cc:492] T f2af8f5d5e784e1988a957666844db6b P b4abc88ed2e54830ae3a317629916117: Bootstrap starting.
I20240502 01:53:21.856773 14665 raft_consensus.cc:729] T f2af8f5d5e784e1988a957666844db6b P 8446b7a7ea67486288d174eea7a3b3c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8446b7a7ea67486288d174eea7a3b3c6, State: Initialized, Role: FOLLOWER
I20240502 01:53:21.857242 14665 consensus_queue.cc:260] T f2af8f5d5e784e1988a957666844db6b P 8446b7a7ea67486288d174eea7a3b3c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } }
I20240502 01:53:21.858220 14665 ts_tablet_manager.cc:1398] T f2af8f5d5e784e1988a957666844db6b P 8446b7a7ea67486288d174eea7a3b3c6: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.003s
I20240502 01:53:21.858563 14665 tablet_bootstrap.cc:492] T 71b0f9fbee82464994252126b2d631bc P 8446b7a7ea67486288d174eea7a3b3c6: Bootstrap starting.
I20240502 01:53:21.859184 14664 tablet_bootstrap.cc:654] T f2af8f5d5e784e1988a957666844db6b P b4abc88ed2e54830ae3a317629916117: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:21.862594 14665 tablet_bootstrap.cc:654] T 71b0f9fbee82464994252126b2d631bc P 8446b7a7ea67486288d174eea7a3b3c6: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:21.866063 14664 tablet_bootstrap.cc:492] T f2af8f5d5e784e1988a957666844db6b P b4abc88ed2e54830ae3a317629916117: No bootstrap required, opened a new log
I20240502 01:53:21.866228 14665 tablet_bootstrap.cc:492] T 71b0f9fbee82464994252126b2d631bc P 8446b7a7ea67486288d174eea7a3b3c6: No bootstrap required, opened a new log
I20240502 01:53:21.866268 14664 ts_tablet_manager.cc:1367] T f2af8f5d5e784e1988a957666844db6b P b4abc88ed2e54830ae3a317629916117: Time spent bootstrapping tablet: real 0.010s	user 0.002s	sys 0.005s
I20240502 01:53:21.866415 14665 ts_tablet_manager.cc:1367] T 71b0f9fbee82464994252126b2d631bc P 8446b7a7ea67486288d174eea7a3b3c6: Time spent bootstrapping tablet: real 0.008s	user 0.005s	sys 0.000s
I20240502 01:53:21.866403 14663 tablet_bootstrap.cc:492] T 71b0f9fbee82464994252126b2d631bc P 5359629f948641f4b56e35ad6b6a15c4: No bootstrap required, opened a new log
I20240502 01:53:21.866588 14663 ts_tablet_manager.cc:1367] T 71b0f9fbee82464994252126b2d631bc P 5359629f948641f4b56e35ad6b6a15c4: Time spent bootstrapping tablet: real 0.014s	user 0.006s	sys 0.003s
I20240502 01:53:21.867134 14664 raft_consensus.cc:348] T f2af8f5d5e784e1988a957666844db6b P b4abc88ed2e54830ae3a317629916117 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } }
I20240502 01:53:21.867381 14664 raft_consensus.cc:374] T f2af8f5d5e784e1988a957666844db6b P b4abc88ed2e54830ae3a317629916117 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:21.867460 14664 raft_consensus.cc:729] T f2af8f5d5e784e1988a957666844db6b P b4abc88ed2e54830ae3a317629916117 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4abc88ed2e54830ae3a317629916117, State: Initialized, Role: FOLLOWER
I20240502 01:53:21.867409 14663 raft_consensus.cc:348] T 71b0f9fbee82464994252126b2d631bc P 5359629f948641f4b56e35ad6b6a15c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } }
I20240502 01:53:21.867677 14663 raft_consensus.cc:374] T 71b0f9fbee82464994252126b2d631bc P 5359629f948641f4b56e35ad6b6a15c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:21.867787 14663 raft_consensus.cc:729] T 71b0f9fbee82464994252126b2d631bc P 5359629f948641f4b56e35ad6b6a15c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5359629f948641f4b56e35ad6b6a15c4, State: Initialized, Role: FOLLOWER
I20240502 01:53:21.867748 14665 raft_consensus.cc:348] T 71b0f9fbee82464994252126b2d631bc P 8446b7a7ea67486288d174eea7a3b3c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } }
I20240502 01:53:21.867724 14664 consensus_queue.cc:260] T f2af8f5d5e784e1988a957666844db6b P b4abc88ed2e54830ae3a317629916117 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } }
I20240502 01:53:21.867956 14665 raft_consensus.cc:374] T 71b0f9fbee82464994252126b2d631bc P 8446b7a7ea67486288d174eea7a3b3c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:21.868046 14665 raft_consensus.cc:729] T 71b0f9fbee82464994252126b2d631bc P 8446b7a7ea67486288d174eea7a3b3c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8446b7a7ea67486288d174eea7a3b3c6, State: Initialized, Role: FOLLOWER
I20240502 01:53:21.868019 14663 consensus_queue.cc:260] T 71b0f9fbee82464994252126b2d631bc P 5359629f948641f4b56e35ad6b6a15c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } }
I20240502 01:53:21.868305 14665 consensus_queue.cc:260] T 71b0f9fbee82464994252126b2d631bc P 8446b7a7ea67486288d174eea7a3b3c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } }
I20240502 01:53:21.868705 14663 ts_tablet_manager.cc:1398] T 71b0f9fbee82464994252126b2d631bc P 5359629f948641f4b56e35ad6b6a15c4: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 01:53:21.868871 14665 ts_tablet_manager.cc:1398] T 71b0f9fbee82464994252126b2d631bc P 8446b7a7ea67486288d174eea7a3b3c6: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 01:53:21.869314 14665 tablet_bootstrap.cc:492] T 930645182ab748d7b864fece1e858b3f P 8446b7a7ea67486288d174eea7a3b3c6: Bootstrap starting.
I20240502 01:53:21.869304 14664 ts_tablet_manager.cc:1398] T f2af8f5d5e784e1988a957666844db6b P b4abc88ed2e54830ae3a317629916117: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.002s
I20240502 01:53:21.870806 14665 tablet_bootstrap.cc:654] T 930645182ab748d7b864fece1e858b3f P 8446b7a7ea67486288d174eea7a3b3c6: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:21.874249 14665 tablet_bootstrap.cc:492] T 930645182ab748d7b864fece1e858b3f P 8446b7a7ea67486288d174eea7a3b3c6: No bootstrap required, opened a new log
I20240502 01:53:21.874437 14665 ts_tablet_manager.cc:1367] T 930645182ab748d7b864fece1e858b3f P 8446b7a7ea67486288d174eea7a3b3c6: Time spent bootstrapping tablet: real 0.005s	user 0.004s	sys 0.000s
I20240502 01:53:21.875188 14665 raft_consensus.cc:348] T 930645182ab748d7b864fece1e858b3f P 8446b7a7ea67486288d174eea7a3b3c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } }
I20240502 01:53:21.875411 14665 raft_consensus.cc:374] T 930645182ab748d7b864fece1e858b3f P 8446b7a7ea67486288d174eea7a3b3c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:21.875495 14665 raft_consensus.cc:729] T 930645182ab748d7b864fece1e858b3f P 8446b7a7ea67486288d174eea7a3b3c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8446b7a7ea67486288d174eea7a3b3c6, State: Initialized, Role: FOLLOWER
I20240502 01:53:21.875777 14665 consensus_queue.cc:260] T 930645182ab748d7b864fece1e858b3f P 8446b7a7ea67486288d174eea7a3b3c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } }
I20240502 01:53:21.876363 14665 ts_tablet_manager.cc:1398] T 930645182ab748d7b864fece1e858b3f P 8446b7a7ea67486288d174eea7a3b3c6: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 01:53:21.920971 14627 raft_consensus.cc:482] T f2af8f5d5e784e1988a957666844db6b P b4abc88ed2e54830ae3a317629916117 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:21.921231 14627 raft_consensus.cc:504] T f2af8f5d5e784e1988a957666844db6b P b4abc88ed2e54830ae3a317629916117 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } }
I20240502 01:53:21.921896 14627 leader_election.cc:288] T f2af8f5d5e784e1988a957666844db6b P b4abc88ed2e54830ae3a317629916117 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8446b7a7ea67486288d174eea7a3b3c6 (127.9.176.194:34851)
I20240502 01:53:21.922515 13604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2af8f5d5e784e1988a957666844db6b" candidate_uuid: "b4abc88ed2e54830ae3a317629916117" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8446b7a7ea67486288d174eea7a3b3c6" is_pre_election: true
I20240502 01:53:21.922765 13604 raft_consensus.cc:2454] T f2af8f5d5e784e1988a957666844db6b P 8446b7a7ea67486288d174eea7a3b3c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4abc88ed2e54830ae3a317629916117 in term 0.
I20240502 01:53:21.923310 13668 leader_election.cc:302] T f2af8f5d5e784e1988a957666844db6b P b4abc88ed2e54830ae3a317629916117 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 8446b7a7ea67486288d174eea7a3b3c6, b4abc88ed2e54830ae3a317629916117; no voters: 
I20240502 01:53:21.923576 14627 raft_consensus.cc:2781] T f2af8f5d5e784e1988a957666844db6b P b4abc88ed2e54830ae3a317629916117 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:21.923710 14627 raft_consensus.cc:482] T f2af8f5d5e784e1988a957666844db6b P b4abc88ed2e54830ae3a317629916117 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:21.923818 14627 raft_consensus.cc:3037] T f2af8f5d5e784e1988a957666844db6b P b4abc88ed2e54830ae3a317629916117 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:21.925859 14627 raft_consensus.cc:504] T f2af8f5d5e784e1988a957666844db6b P b4abc88ed2e54830ae3a317629916117 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } }
I20240502 01:53:21.926374 14627 leader_election.cc:288] T f2af8f5d5e784e1988a957666844db6b P b4abc88ed2e54830ae3a317629916117 [CANDIDATE]: Term 1 election: Requested vote from peers 8446b7a7ea67486288d174eea7a3b3c6 (127.9.176.194:34851)
I20240502 01:53:21.926961 13604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2af8f5d5e784e1988a957666844db6b" candidate_uuid: "b4abc88ed2e54830ae3a317629916117" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8446b7a7ea67486288d174eea7a3b3c6"
I20240502 01:53:21.927196 13604 raft_consensus.cc:3037] T f2af8f5d5e784e1988a957666844db6b P 8446b7a7ea67486288d174eea7a3b3c6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:21.929203 13604 raft_consensus.cc:2454] T f2af8f5d5e784e1988a957666844db6b P 8446b7a7ea67486288d174eea7a3b3c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4abc88ed2e54830ae3a317629916117 in term 1.
I20240502 01:53:21.929666 13668 leader_election.cc:302] T f2af8f5d5e784e1988a957666844db6b P b4abc88ed2e54830ae3a317629916117 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 8446b7a7ea67486288d174eea7a3b3c6, b4abc88ed2e54830ae3a317629916117; no voters: 
I20240502 01:53:21.929975 14627 raft_consensus.cc:2781] T f2af8f5d5e784e1988a957666844db6b P b4abc88ed2e54830ae3a317629916117 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:21.930145 14627 raft_consensus.cc:686] T f2af8f5d5e784e1988a957666844db6b P b4abc88ed2e54830ae3a317629916117 [term 1 LEADER]: Becoming Leader. State: Replica: b4abc88ed2e54830ae3a317629916117, State: Running, Role: LEADER
I20240502 01:53:21.930425 14627 consensus_queue.cc:237] T f2af8f5d5e784e1988a957666844db6b P b4abc88ed2e54830ae3a317629916117 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } }
I20240502 01:53:21.933292 13323 catalog_manager.cc:5484] T f2af8f5d5e784e1988a957666844db6b P b4abc88ed2e54830ae3a317629916117 reported cstate change: term changed from 0 to 1, leader changed from <none> to b4abc88ed2e54830ae3a317629916117 (127.9.176.195). New cstate: current_term: 1 leader_uuid: "b4abc88ed2e54830ae3a317629916117" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } health_report { overall_health: HEALTHY } } }
I20240502 01:53:21.941291 14534 raft_consensus.cc:482] T 930645182ab748d7b864fece1e858b3f P 8446b7a7ea67486288d174eea7a3b3c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:21.941540 14534 raft_consensus.cc:504] T 930645182ab748d7b864fece1e858b3f P 8446b7a7ea67486288d174eea7a3b3c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } }
I20240502 01:53:21.942176 14534 leader_election.cc:288] T 930645182ab748d7b864fece1e858b3f P 8446b7a7ea67486288d174eea7a3b3c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b4abc88ed2e54830ae3a317629916117 (127.9.176.195:41211)
I20240502 01:53:21.942844 13734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "930645182ab748d7b864fece1e858b3f" candidate_uuid: "8446b7a7ea67486288d174eea7a3b3c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4abc88ed2e54830ae3a317629916117" is_pre_election: true
I20240502 01:53:21.943081 13734 raft_consensus.cc:2454] T 930645182ab748d7b864fece1e858b3f P b4abc88ed2e54830ae3a317629916117 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8446b7a7ea67486288d174eea7a3b3c6 in term 0.
I20240502 01:53:21.943532 13538 leader_election.cc:302] T 930645182ab748d7b864fece1e858b3f P 8446b7a7ea67486288d174eea7a3b3c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 8446b7a7ea67486288d174eea7a3b3c6, b4abc88ed2e54830ae3a317629916117; no voters: 
I20240502 01:53:21.943852 14534 raft_consensus.cc:2781] T 930645182ab748d7b864fece1e858b3f P 8446b7a7ea67486288d174eea7a3b3c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:21.943969 14534 raft_consensus.cc:482] T 930645182ab748d7b864fece1e858b3f P 8446b7a7ea67486288d174eea7a3b3c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:21.944041 14534 raft_consensus.cc:3037] T 930645182ab748d7b864fece1e858b3f P 8446b7a7ea67486288d174eea7a3b3c6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:21.945994 14534 raft_consensus.cc:504] T 930645182ab748d7b864fece1e858b3f P 8446b7a7ea67486288d174eea7a3b3c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } }
I20240502 01:53:21.946475 14534 leader_election.cc:288] T 930645182ab748d7b864fece1e858b3f P 8446b7a7ea67486288d174eea7a3b3c6 [CANDIDATE]: Term 1 election: Requested vote from peers b4abc88ed2e54830ae3a317629916117 (127.9.176.195:41211)
I20240502 01:53:21.947125 13734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "930645182ab748d7b864fece1e858b3f" candidate_uuid: "8446b7a7ea67486288d174eea7a3b3c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4abc88ed2e54830ae3a317629916117"
I20240502 01:53:21.947347 13734 raft_consensus.cc:3037] T 930645182ab748d7b864fece1e858b3f P b4abc88ed2e54830ae3a317629916117 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:21.949410 13734 raft_consensus.cc:2454] T 930645182ab748d7b864fece1e858b3f P b4abc88ed2e54830ae3a317629916117 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8446b7a7ea67486288d174eea7a3b3c6 in term 1.
I20240502 01:53:21.949947 13538 leader_election.cc:302] T 930645182ab748d7b864fece1e858b3f P 8446b7a7ea67486288d174eea7a3b3c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 8446b7a7ea67486288d174eea7a3b3c6, b4abc88ed2e54830ae3a317629916117; no voters: 
I20240502 01:53:21.950309 14534 raft_consensus.cc:2781] T 930645182ab748d7b864fece1e858b3f P 8446b7a7ea67486288d174eea7a3b3c6 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:21.950735 14534 raft_consensus.cc:686] T 930645182ab748d7b864fece1e858b3f P 8446b7a7ea67486288d174eea7a3b3c6 [term 1 LEADER]: Becoming Leader. State: Replica: 8446b7a7ea67486288d174eea7a3b3c6, State: Running, Role: LEADER
I20240502 01:53:21.951033 14534 consensus_queue.cc:237] T 930645182ab748d7b864fece1e858b3f P 8446b7a7ea67486288d174eea7a3b3c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } }
I20240502 01:53:21.953987 13323 catalog_manager.cc:5484] T 930645182ab748d7b864fece1e858b3f P 8446b7a7ea67486288d174eea7a3b3c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8446b7a7ea67486288d174eea7a3b3c6 (127.9.176.194). New cstate: current_term: 1 leader_uuid: "8446b7a7ea67486288d174eea7a3b3c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:22.152225 14489 raft_consensus.cc:482] T 71b0f9fbee82464994252126b2d631bc P 5359629f948641f4b56e35ad6b6a15c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:22.152446 14489 raft_consensus.cc:504] T 71b0f9fbee82464994252126b2d631bc P 5359629f948641f4b56e35ad6b6a15c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } }
I20240502 01:53:22.153013 14489 leader_election.cc:288] T 71b0f9fbee82464994252126b2d631bc P 5359629f948641f4b56e35ad6b6a15c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8446b7a7ea67486288d174eea7a3b3c6 (127.9.176.194:34851)
I20240502 01:53:22.153621 13604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71b0f9fbee82464994252126b2d631bc" candidate_uuid: "5359629f948641f4b56e35ad6b6a15c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8446b7a7ea67486288d174eea7a3b3c6" is_pre_election: true
I20240502 01:53:22.153849 13604 raft_consensus.cc:2454] T 71b0f9fbee82464994252126b2d631bc P 8446b7a7ea67486288d174eea7a3b3c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5359629f948641f4b56e35ad6b6a15c4 in term 0.
I20240502 01:53:22.154296 13402 leader_election.cc:302] T 71b0f9fbee82464994252126b2d631bc P 5359629f948641f4b56e35ad6b6a15c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 5359629f948641f4b56e35ad6b6a15c4, 8446b7a7ea67486288d174eea7a3b3c6; no voters: 
I20240502 01:53:22.154587 14489 raft_consensus.cc:2781] T 71b0f9fbee82464994252126b2d631bc P 5359629f948641f4b56e35ad6b6a15c4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:22.154726 14489 raft_consensus.cc:482] T 71b0f9fbee82464994252126b2d631bc P 5359629f948641f4b56e35ad6b6a15c4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:22.154798 14489 raft_consensus.cc:3037] T 71b0f9fbee82464994252126b2d631bc P 5359629f948641f4b56e35ad6b6a15c4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:22.156674 14489 raft_consensus.cc:504] T 71b0f9fbee82464994252126b2d631bc P 5359629f948641f4b56e35ad6b6a15c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } }
I20240502 01:53:22.157122 14489 leader_election.cc:288] T 71b0f9fbee82464994252126b2d631bc P 5359629f948641f4b56e35ad6b6a15c4 [CANDIDATE]: Term 1 election: Requested vote from peers 8446b7a7ea67486288d174eea7a3b3c6 (127.9.176.194:34851)
I20240502 01:53:22.157724 13604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71b0f9fbee82464994252126b2d631bc" candidate_uuid: "5359629f948641f4b56e35ad6b6a15c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8446b7a7ea67486288d174eea7a3b3c6"
I20240502 01:53:22.157910 13604 raft_consensus.cc:3037] T 71b0f9fbee82464994252126b2d631bc P 8446b7a7ea67486288d174eea7a3b3c6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:22.160068 13604 raft_consensus.cc:2454] T 71b0f9fbee82464994252126b2d631bc P 8446b7a7ea67486288d174eea7a3b3c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5359629f948641f4b56e35ad6b6a15c4 in term 1.
I20240502 01:53:22.160590 13402 leader_election.cc:302] T 71b0f9fbee82464994252126b2d631bc P 5359629f948641f4b56e35ad6b6a15c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 5359629f948641f4b56e35ad6b6a15c4, 8446b7a7ea67486288d174eea7a3b3c6; no voters: 
I20240502 01:53:22.160928 14489 raft_consensus.cc:2781] T 71b0f9fbee82464994252126b2d631bc P 5359629f948641f4b56e35ad6b6a15c4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:22.161072 14489 raft_consensus.cc:686] T 71b0f9fbee82464994252126b2d631bc P 5359629f948641f4b56e35ad6b6a15c4 [term 1 LEADER]: Becoming Leader. State: Replica: 5359629f948641f4b56e35ad6b6a15c4, State: Running, Role: LEADER
I20240502 01:53:22.161353 14489 consensus_queue.cc:237] T 71b0f9fbee82464994252126b2d631bc P 5359629f948641f4b56e35ad6b6a15c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } }
I20240502 01:53:22.164172 13323 catalog_manager.cc:5484] T 71b0f9fbee82464994252126b2d631bc P 5359629f948641f4b56e35ad6b6a15c4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5359629f948641f4b56e35ad6b6a15c4 (127.9.176.193). New cstate: current_term: 1 leader_uuid: "5359629f948641f4b56e35ad6b6a15c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:22.347296 14534 consensus_queue.cc:1035] T 930645182ab748d7b864fece1e858b3f P 8446b7a7ea67486288d174eea7a3b3c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:22.503743 14627 consensus_queue.cc:1035] T f2af8f5d5e784e1988a957666844db6b P b4abc88ed2e54830ae3a317629916117 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:53:22.580507 14678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:22.580870 14678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:22.588797 14678 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240502 01:53:22.770200 14489 consensus_queue.cc:1035] T 71b0f9fbee82464994252126b2d631bc P 5359629f948641f4b56e35ad6b6a15c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 245.087 ms
  time per row: 0.122543 ms
I20240502 01:53:22.980289  9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
/tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.196:41329
--local_ip_for_outbound_sockets=127.9.176.196
--tserver_master_addrs=127.9.176.254:44447
--webserver_port=34361
--webserver_interface=127.9.176.196
--builtin_ntp_servers=127.9.176.212:33337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 01:53:23.321369 14736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:23.321705 14736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:53:23.321870 14736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:23.329798 14736 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:53:23.329960 14736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:53:23.330118 14736 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.196
I20240502 01:53:23.339805 14736 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:33337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.196:41329
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.9.176.196
--webserver_port=34361
--tserver_master_addrs=127.9.176.254:44447
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.196
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:53:23.341156 14736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:53:23.343101 14736 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:53:23.356025 14742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:23.356699 14744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:23.357216 14741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:53:23.357969 14736 server_base.cc:1034] running on GCE node
I20240502 01:53:23.358714 14736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:53:23.360126 14736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:53:23.361327 14736 hybrid_clock.cc:648] HybridClock initialized: now 1714614803361278 us; error 39 us; skew 500 ppm
I20240502 01:53:23.361713 14736 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:53:23.370386 14736 webserver.cc:460] Webserver started at http://127.9.176.196:34361/ using document root <none> and password file <none>
I20240502 01:53:23.371232 14736 fs_manager.cc:359] Metadata directory not provided
I20240502 01:53:23.371371 14736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:53:23.377166 14736 fs_manager.cc:709] Time spent opening directory manager: real 0.004s	user 0.004s	sys 0.000s
I20240502 01:53:23.380268 14750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:53:23.381676 14736 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20240502 01:53:23.381844 14736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "ce90a4bd06ba41dd8e5aa227de9780a5"
format_stamp: "Formatted at 2024-05-02 01:53:11 on dist-test-slave-j3hh"
I20240502 01:53:23.382774 14736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:53:23.402604 14736 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:53:23.403528 14736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:53:23.403899 14736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:53:23.404944 14736 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:53:23.407114 14757 ts_tablet_manager.cc:522] Loading tablet metadata (0/1 complete)
I20240502 01:53:23.413189 14736 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240502 01:53:23.413321 14736 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.007s	user 0.001s	sys 0.000s
I20240502 01:53:23.413461 14736 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240502 01:53:23.416589 14736 ts_tablet_manager.cc:590] Registered 1 tablets
I20240502 01:53:23.416698 14736 ts_tablet_manager.cc:569] Time spent register tablets: real 0.003s	user 0.003s	sys 0.000s
I20240502 01:53:23.417232 14757 tablet_bootstrap.cc:492] T d38a05f7ffa94cc5abc50a76a83692b5 P ce90a4bd06ba41dd8e5aa227de9780a5: Bootstrap starting.
I20240502 01:53:23.437963 14757 log.cc:824] T d38a05f7ffa94cc5abc50a76a83692b5 P ce90a4bd06ba41dd8e5aa227de9780a5: Log is configured to *not* fsync() on all Append() calls
I20240502 01:53:23.442741 14757 tablet_bootstrap.cc:492] T d38a05f7ffa94cc5abc50a76a83692b5 P ce90a4bd06ba41dd8e5aa227de9780a5: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 01:53:23.443769 14757 tablet_bootstrap.cc:492] T d38a05f7ffa94cc5abc50a76a83692b5 P ce90a4bd06ba41dd8e5aa227de9780a5: Bootstrap complete.
I20240502 01:53:23.445410 14757 ts_tablet_manager.cc:1367] T d38a05f7ffa94cc5abc50a76a83692b5 P ce90a4bd06ba41dd8e5aa227de9780a5: Time spent bootstrapping tablet: real 0.029s	user 0.009s	sys 0.017s
I20240502 01:53:23.455281 14757 raft_consensus.cc:348] T d38a05f7ffa94cc5abc50a76a83692b5 P ce90a4bd06ba41dd8e5aa227de9780a5 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 41329 } }
I20240502 01:53:23.456991 14757 raft_consensus.cc:729] T d38a05f7ffa94cc5abc50a76a83692b5 P ce90a4bd06ba41dd8e5aa227de9780a5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce90a4bd06ba41dd8e5aa227de9780a5, State: Initialized, Role: FOLLOWER
I20240502 01:53:23.457885 14757 consensus_queue.cc:260] T d38a05f7ffa94cc5abc50a76a83692b5 P ce90a4bd06ba41dd8e5aa227de9780a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 41329 } }
I20240502 01:53:23.464298 14757 ts_tablet_manager.cc:1398] T d38a05f7ffa94cc5abc50a76a83692b5 P ce90a4bd06ba41dd8e5aa227de9780a5: Time spent starting tablet: real 0.018s	user 0.008s	sys 0.009s
I20240502 01:53:23.471100 14736 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.196:41329
I20240502 01:53:23.471171 14866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.196:41329 every 8 connection(s)
I20240502 01:53:23.472568 14736 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/data/info.pb
W20240502 01:53:23.473846 14868 tablet.cc:2369] T d38a05f7ffa94cc5abc50a76a83692b5 P ce90a4bd06ba41dd8e5aa227de9780a5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:53:23.475206  9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu as pid 14736
I20240502 01:53:23.476023  9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
/tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.197:35757
--local_ip_for_outbound_sockets=127.9.176.197
--tserver_master_addrs=127.9.176.254:44447
--webserver_port=36841
--webserver_interface=127.9.176.197
--builtin_ntp_servers=127.9.176.212:33337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:53:23.488965 14867 heartbeater.cc:347] Connected to a master server at 127.9.176.254:44447
I20240502 01:53:23.489300 14867 heartbeater.cc:464] Registering TS with master...
I20240502 01:53:23.490192 14867 heartbeater.cc:511] Master 127.9.176.254:44447 requested a full tablet report, sending...
I20240502 01:53:23.492892 13318 ts_manager.cc:194] Re-registered known tserver with Master: ce90a4bd06ba41dd8e5aa227de9780a5 (127.9.176.196:41329)
I20240502 01:53:23.494719 13318 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.176.196:42315
W20240502 01:53:23.825089 14871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:23.825390 14871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:53:23.825510 14871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:23.833220 14871 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:53:23.833343 14871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:53:23.833462 14871 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.197
I20240502 01:53:23.842784 14871 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:33337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.197:35757
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.9.176.197
--webserver_port=36841
--tserver_master_addrs=127.9.176.254:44447
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.197
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:53:23.844319 14871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:53:23.846195 14871 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:53:23.858110 14876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:23.858743 14879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:23.859326 14877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:53:23.859990 14871 server_base.cc:1034] running on GCE node
I20240502 01:53:23.860599 14871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:53:23.861534 14871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:53:23.862735 14871 hybrid_clock.cc:648] HybridClock initialized: now 1714614803862698 us; error 63 us; skew 500 ppm
I20240502 01:53:23.863087 14871 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:53:23.865463 14871 webserver.cc:460] Webserver started at http://127.9.176.197:36841/ using document root <none> and password file <none>
I20240502 01:53:23.866195 14871 fs_manager.cc:359] Metadata directory not provided
I20240502 01:53:23.866323 14871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:53:23.871665 14871 fs_manager.cc:709] Time spent opening directory manager: real 0.003s	user 0.004s	sys 0.000s
I20240502 01:53:23.874389 14886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:53:23.875738 14871 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240502 01:53:23.875909 14871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "7e74a3d5df6a483c95c9dabfc99577bc"
format_stamp: "Formatted at 2024-05-02 01:53:11 on dist-test-slave-j3hh"
I20240502 01:53:23.876761 14871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:53:23.900002 14871 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:53:23.900812 14871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:53:23.901146 14871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:53:23.902369 14871 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:53:23.904162 14871 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:53:23.904247 14871 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:53:23.904386 14871 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:53:23.904443 14871 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:53:23.935515 14871 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.197:35757
I20240502 01:53:23.935581 15000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.197:35757 every 8 connection(s)
I20240502 01:53:23.937044 14871 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 01:53:23.945255  9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu as pid 14871
I20240502 01:53:23.945852  9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
/tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.198:38527
--local_ip_for_outbound_sockets=127.9.176.198
--tserver_master_addrs=127.9.176.254:44447
--webserver_port=37743
--webserver_interface=127.9.176.198
--builtin_ntp_servers=127.9.176.212:33337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:53:23.949724 15001 heartbeater.cc:347] Connected to a master server at 127.9.176.254:44447
I20240502 01:53:23.950018 15001 heartbeater.cc:464] Registering TS with master...
I20240502 01:53:23.950834 15001 heartbeater.cc:511] Master 127.9.176.254:44447 requested a full tablet report, sending...
I20240502 01:53:23.952207 13322 ts_manager.cc:194] Re-registered known tserver with Master: 7e74a3d5df6a483c95c9dabfc99577bc (127.9.176.197:35757)
I20240502 01:53:23.953429 13322 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.176.197:44471
W20240502 01:53:24.273016 15004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:24.273324 15004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:53:24.273439 15004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:24.280660 15004 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:53:24.280786 15004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:53:24.280936 15004 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.198
I20240502 01:53:24.290050 15004 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:33337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.198:38527
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.9.176.198
--webserver_port=37743
--tserver_master_addrs=127.9.176.254:44447
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.198
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:53:24.291405 15004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:53:24.293097 15004 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:53:24.304486 15009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:24.304970 15010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:24.305526 15012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:53:24.306288 15004 server_base.cc:1034] running on GCE node
I20240502 01:53:24.306903 15004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:53:24.307873 15004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:53:24.309087 15004 hybrid_clock.cc:648] HybridClock initialized: now 1714614804309065 us; error 44 us; skew 500 ppm
I20240502 01:53:24.309388 15004 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:53:24.311419 15004 webserver.cc:460] Webserver started at http://127.9.176.198:37743/ using document root <none> and password file <none>
I20240502 01:53:24.312214 15004 fs_manager.cc:359] Metadata directory not provided
I20240502 01:53:24.312335 15004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:53:24.317766 15004 fs_manager.cc:709] Time spent opening directory manager: real 0.004s	user 0.005s	sys 0.000s
I20240502 01:53:24.320634 15018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:53:24.322032 15004 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240502 01:53:24.322219 15004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f"
format_stamp: "Formatted at 2024-05-02 01:53:12 on dist-test-slave-j3hh"
I20240502 01:53:24.323084 15004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:53:24.351857 15004 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:53:24.352756 15004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:53:24.353037 15004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:53:24.354013 15004 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:53:24.355556 15004 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:53:24.355661 15004 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:53:24.355793 15004 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:53:24.355844 15004 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:53:24.389770 15004 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.198:38527
I20240502 01:53:24.389817 15132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.198:38527 every 8 connection(s)
I20240502 01:53:24.391286 15004 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 01:53:24.399952  9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu as pid 15004
I20240502 01:53:24.400532  9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
/tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.199:38645
--local_ip_for_outbound_sockets=127.9.176.199
--tserver_master_addrs=127.9.176.254:44447
--webserver_port=35747
--webserver_interface=127.9.176.199
--builtin_ntp_servers=127.9.176.212:33337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:53:24.404186 15133 heartbeater.cc:347] Connected to a master server at 127.9.176.254:44447
I20240502 01:53:24.404502 15133 heartbeater.cc:464] Registering TS with master...
I20240502 01:53:24.405314 15133 heartbeater.cc:511] Master 127.9.176.254:44447 requested a full tablet report, sending...
I20240502 01:53:24.406780 13322 ts_manager.cc:194] Re-registered known tserver with Master: 9e4fa889fab047baa5bd3dcd77b2ca8f (127.9.176.198:38527)
I20240502 01:53:24.408051 13322 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.176.198:34065
I20240502 01:53:24.497741 14867 heartbeater.cc:503] Master 127.9.176.254:44447 was elected leader, sending a full tablet report...
W20240502 01:53:24.741652 15136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:24.742017 15136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:53:24.742163 15136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:24.749976 15136 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:53:24.750125 15136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:53:24.750272 15136 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.199
I20240502 01:53:24.759230 15136 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:33337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.199:38645
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.9.176.199
--webserver_port=35747
--tserver_master_addrs=127.9.176.254:44447
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.199
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:53:24.761052 15136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:53:24.763604 15136 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:53:24.777058 15141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:24.777770 15144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:24.778316 15142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:53:24.779062 15136 server_base.cc:1034] running on GCE node
I20240502 01:53:24.779657 15136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:53:24.780627 15136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:53:24.781848 15136 hybrid_clock.cc:648] HybridClock initialized: now 1714614804781784 us; error 37 us; skew 500 ppm
I20240502 01:53:24.782137 15136 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:53:24.784315 15136 webserver.cc:460] Webserver started at http://127.9.176.199:35747/ using document root <none> and password file <none>
I20240502 01:53:24.784890 15149 raft_consensus.cc:482] T d38a05f7ffa94cc5abc50a76a83692b5 P ce90a4bd06ba41dd8e5aa227de9780a5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:24.785284 15136 fs_manager.cc:359] Metadata directory not provided
I20240502 01:53:24.785176 15149 raft_consensus.cc:504] T d38a05f7ffa94cc5abc50a76a83692b5 P ce90a4bd06ba41dd8e5aa227de9780a5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 41329 } }
I20240502 01:53:24.785436 15136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:53:24.786828 15149 leader_election.cc:288] T d38a05f7ffa94cc5abc50a76a83692b5 P ce90a4bd06ba41dd8e5aa227de9780a5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5359629f948641f4b56e35ad6b6a15c4 (127.9.176.193:33849)
I20240502 01:53:24.792156 15136 fs_manager.cc:709] Time spent opening directory manager: real 0.004s	user 0.002s	sys 0.001s
I20240502 01:53:24.794366 13471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d38a05f7ffa94cc5abc50a76a83692b5" candidate_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5359629f948641f4b56e35ad6b6a15c4" is_pre_election: true
I20240502 01:53:24.794615 13471 raft_consensus.cc:2454] T d38a05f7ffa94cc5abc50a76a83692b5 P 5359629f948641f4b56e35ad6b6a15c4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce90a4bd06ba41dd8e5aa227de9780a5 in term 1.
I20240502 01:53:24.795223 14754 leader_election.cc:302] T d38a05f7ffa94cc5abc50a76a83692b5 P ce90a4bd06ba41dd8e5aa227de9780a5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 5359629f948641f4b56e35ad6b6a15c4, ce90a4bd06ba41dd8e5aa227de9780a5; no voters: 
I20240502 01:53:24.795559 15154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:53:24.795652 15149 raft_consensus.cc:2781] T d38a05f7ffa94cc5abc50a76a83692b5 P ce90a4bd06ba41dd8e5aa227de9780a5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 01:53:24.795818 15149 raft_consensus.cc:482] T d38a05f7ffa94cc5abc50a76a83692b5 P ce90a4bd06ba41dd8e5aa227de9780a5 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:24.795920 15149 raft_consensus.cc:3037] T d38a05f7ffa94cc5abc50a76a83692b5 P ce90a4bd06ba41dd8e5aa227de9780a5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:53:24.796953 15136 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20240502 01:53:24.797097 15136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "7505fefeaf804ed8aeb3672f386ab959"
format_stamp: "Formatted at 2024-05-02 01:53:12 on dist-test-slave-j3hh"
I20240502 01:53:24.797847 15136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:53:24.799394 15149 raft_consensus.cc:504] T d38a05f7ffa94cc5abc50a76a83692b5 P ce90a4bd06ba41dd8e5aa227de9780a5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 41329 } }
I20240502 01:53:24.799985 15149 leader_election.cc:288] T d38a05f7ffa94cc5abc50a76a83692b5 P ce90a4bd06ba41dd8e5aa227de9780a5 [CANDIDATE]: Term 2 election: Requested vote from peers 5359629f948641f4b56e35ad6b6a15c4 (127.9.176.193:33849)
I20240502 01:53:24.800544 13471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d38a05f7ffa94cc5abc50a76a83692b5" candidate_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5359629f948641f4b56e35ad6b6a15c4"
I20240502 01:53:24.800791 13471 raft_consensus.cc:3037] T d38a05f7ffa94cc5abc50a76a83692b5 P 5359629f948641f4b56e35ad6b6a15c4 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:53:24.803678 13471 raft_consensus.cc:2454] T d38a05f7ffa94cc5abc50a76a83692b5 P 5359629f948641f4b56e35ad6b6a15c4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce90a4bd06ba41dd8e5aa227de9780a5 in term 2.
I20240502 01:53:24.804162 14754 leader_election.cc:302] T d38a05f7ffa94cc5abc50a76a83692b5 P ce90a4bd06ba41dd8e5aa227de9780a5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 5359629f948641f4b56e35ad6b6a15c4, ce90a4bd06ba41dd8e5aa227de9780a5; no voters: 
I20240502 01:53:24.804476 15149 raft_consensus.cc:2781] T d38a05f7ffa94cc5abc50a76a83692b5 P ce90a4bd06ba41dd8e5aa227de9780a5 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:53:24.804941 15149 raft_consensus.cc:686] T d38a05f7ffa94cc5abc50a76a83692b5 P ce90a4bd06ba41dd8e5aa227de9780a5 [term 2 LEADER]: Becoming Leader. State: Replica: ce90a4bd06ba41dd8e5aa227de9780a5, State: Running, Role: LEADER
I20240502 01:53:24.805423 15149 consensus_queue.cc:237] T d38a05f7ffa94cc5abc50a76a83692b5 P ce90a4bd06ba41dd8e5aa227de9780a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 41329 } }
I20240502 01:53:24.809799 13322 catalog_manager.cc:5484] T d38a05f7ffa94cc5abc50a76a83692b5 P ce90a4bd06ba41dd8e5aa227de9780a5 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 41329 } health_report { overall_health: HEALTHY } } }
I20240502 01:53:24.823694 15136 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:53:24.824565 15136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:53:24.824851 15136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:53:24.825878 15136 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:53:24.827595 15136 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:53:24.827680 15136 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:53:24.827803 15136 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:53:24.827849 15136 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:53:24.862465 15136 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.199:38645
I20240502 01:53:24.862501 15272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.199:38645 every 8 connection(s)
I20240502 01:53:24.864015 15136 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 01:53:24.866391  9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu as pid 15136
I20240502 01:53:24.866770  9923 test_util.cc:268] Using random seed: -564205910
I20240502 01:53:24.881846 15273 heartbeater.cc:347] Connected to a master server at 127.9.176.254:44447
I20240502 01:53:24.882184 15273 heartbeater.cc:464] Registering TS with master...
I20240502 01:53:24.883219 15273 heartbeater.cc:511] Master 127.9.176.254:44447 requested a full tablet report, sending...
I20240502 01:53:24.884764 13322 ts_manager.cc:194] Re-registered known tserver with Master: 7505fefeaf804ed8aeb3672f386ab959 (127.9.176.199:38645)
I20240502 01:53:24.885583 13322 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.176.199:59251
I20240502 01:53:24.887219  9923 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_0 already exists
I20240502 01:53:24.887820  9923 test_util.cc:268] Using random seed: -564184869
I20240502 01:53:24.902838  9923 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_1 already exists
I20240502 01:53:24.903306  9923 test_util.cc:268] Using random seed: -564169406
I20240502 01:53:24.926769  9923 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_2 already exists
I20240502 01:53:24.929059  9923 test_util.cc:268] Using random seed: -564143666
I20240502 01:53:24.955993 15001 heartbeater.cc:503] Master 127.9.176.254:44447 was elected leader, sending a full tablet report...
I20240502 01:53:25.021164  9923 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_3 already exists
I20240502 01:53:25.032146  9923 test_util.cc:268] Using random seed: -564040576
I20240502 01:53:25.152782  9923 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_4 already exists
I20240502 01:53:25.371465 13471 raft_consensus.cc:1261] T d38a05f7ffa94cc5abc50a76a83692b5 P 5359629f948641f4b56e35ad6b6a15c4 [term 2 FOLLOWER]: Refusing update from remote peer ce90a4bd06ba41dd8e5aa227de9780a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 01:53:25.380286 15372 consensus_queue.cc:1035] T d38a05f7ffa94cc5abc50a76a83692b5 P ce90a4bd06ba41dd8e5aa227de9780a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:25.411262 15133 heartbeater.cc:503] Master 127.9.176.254:44447 was elected leader, sending a full tablet report...
W20240502 01:53:25.582755 14754 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.176.193:33849, user_credentials={real_user=slave}} blocked reactor thread for 58882.6us
I20240502 01:53:25.888861 15273 heartbeater.cc:503] Master 127.9.176.254:44447 was elected leader, sending a full tablet report...
W20240502 01:53:25.909653 13538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.176.195:41211, user_credentials={real_user=slave}} blocked reactor thread for 81477.8us
I20240502 01:53:26.662469 13451 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:53:26.694275 14934 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:26.710273 15208 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:26.817152 15067 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:26.834733 14799 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:53:26.961799 13538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.176.195:41211, user_credentials={real_user=slave}} blocked reactor thread for 55076.8us
I20240502 01:53:26.971616 13584 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:53:26.978415 13715 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:53:27.846963 14934 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:27.881742 13715 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:53:27.887253 13584 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:53:27.954607 14799 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:28.011234 15208 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:28.013705 13451 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:53:28.030330 15067 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:28.216674 13735 consensus_queue.cc:237] T 4c0ab7c2995d4ccfa170f365b4e09b1d P b4abc88ed2e54830ae3a317629916117 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 52, Committed index: 52, Last appended: 1.52, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 53 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } attrs { replace: true } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } } peers { permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: true } }
I20240502 01:53:28.220539 13470 raft_consensus.cc:1261] T 4c0ab7c2995d4ccfa170f365b4e09b1d P 5359629f948641f4b56e35ad6b6a15c4 [term 1 FOLLOWER]: Refusing update from remote peer b4abc88ed2e54830ae3a317629916117: Log matching property violated. Preceding OpId in replica: term: 1 index: 52. Preceding OpId from leader: term: 1 index: 53. (index mismatch)
I20240502 01:53:28.224198 14683 consensus_queue.cc:1035] T 4c0ab7c2995d4ccfa170f365b4e09b1d P b4abc88ed2e54830ae3a317629916117 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 53, Last known committed idx: 52, Time since last communication: 0.000s
I20240502 01:53:28.228746 15404 raft_consensus.cc:2932] T 4c0ab7c2995d4ccfa170f365b4e09b1d P b4abc88ed2e54830ae3a317629916117 [term 1 LEADER]: Committing config change with OpId 1.53: config changed from index -1 to 53, NON_VOTER 7e74a3d5df6a483c95c9dabfc99577bc (127.9.176.197) added. New config: { opid_index: 53 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } attrs { replace: true } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } } peers { permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: true } } }
I20240502 01:53:28.239557 13323 catalog_manager.cc:5484] T 4c0ab7c2995d4ccfa170f365b4e09b1d P b4abc88ed2e54830ae3a317629916117 reported cstate change: config changed from index -1 to 53, NON_VOTER 7e74a3d5df6a483c95c9dabfc99577bc (127.9.176.197) added. New cstate: current_term: 1 leader_uuid: "b4abc88ed2e54830ae3a317629916117" committed_config { opid_index: 53 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 01:53:28.249882 13668 consensus_peers.cc:485] T 4c0ab7c2995d4ccfa170f365b4e09b1d P b4abc88ed2e54830ae3a317629916117 -> Peer 7e74a3d5df6a483c95c9dabfc99577bc (127.9.176.197:35757): Couldn't send request to peer 7e74a3d5df6a483c95c9dabfc99577bc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4c0ab7c2995d4ccfa170f365b4e09b1d. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:28.250771 13471 raft_consensus.cc:2932] T 4c0ab7c2995d4ccfa170f365b4e09b1d P 5359629f948641f4b56e35ad6b6a15c4 [term 1 FOLLOWER]: Committing config change with OpId 1.53: config changed from index -1 to 53, NON_VOTER 7e74a3d5df6a483c95c9dabfc99577bc (127.9.176.197) added. New config: { opid_index: 53 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } attrs { replace: true } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } } peers { permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: true } } }
I20240502 01:53:28.341990 13735 consensus_queue.cc:237] T 57765e8e3ff7438ca9e588db8e685a03 P b4abc88ed2e54830ae3a317629916117 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 65, Committed index: 65, Last appended: 1.65, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } attrs { replace: true } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: true } }
I20240502 01:53:28.352041 13601 raft_consensus.cc:1261] T 57765e8e3ff7438ca9e588db8e685a03 P 8446b7a7ea67486288d174eea7a3b3c6 [term 1 FOLLOWER]: Refusing update from remote peer b4abc88ed2e54830ae3a317629916117: Log matching property violated. Preceding OpId in replica: term: 1 index: 65. Preceding OpId from leader: term: 1 index: 66. (index mismatch)
I20240502 01:53:28.353466 15504 consensus_queue.cc:1035] T 57765e8e3ff7438ca9e588db8e685a03 P b4abc88ed2e54830ae3a317629916117 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 66, Last known committed idx: 65, Time since last communication: 0.000s
I20240502 01:53:28.356909 15369 raft_consensus.cc:2932] T 57765e8e3ff7438ca9e588db8e685a03 P b4abc88ed2e54830ae3a317629916117 [term 1 LEADER]: Committing config change with OpId 1.66: config changed from index -1 to 66, NON_VOTER 9e4fa889fab047baa5bd3dcd77b2ca8f (127.9.176.198) added. New config: { opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } attrs { replace: true } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: true } } }
W20240502 01:53:28.359045 13671 consensus_peers.cc:485] T 57765e8e3ff7438ca9e588db8e685a03 P b4abc88ed2e54830ae3a317629916117 -> Peer 9e4fa889fab047baa5bd3dcd77b2ca8f (127.9.176.198:38527): Couldn't send request to peer 9e4fa889fab047baa5bd3dcd77b2ca8f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 57765e8e3ff7438ca9e588db8e685a03. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:28.363608 13603 raft_consensus.cc:2932] T 57765e8e3ff7438ca9e588db8e685a03 P 8446b7a7ea67486288d174eea7a3b3c6 [term 1 FOLLOWER]: Committing config change with OpId 1.66: config changed from index -1 to 66, NON_VOTER 9e4fa889fab047baa5bd3dcd77b2ca8f (127.9.176.198) added. New config: { opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } attrs { replace: true } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: true } } }
I20240502 01:53:28.372669 13318 catalog_manager.cc:5484] T 57765e8e3ff7438ca9e588db8e685a03 P b4abc88ed2e54830ae3a317629916117 reported cstate change: config changed from index -1 to 66, NON_VOTER 9e4fa889fab047baa5bd3dcd77b2ca8f (127.9.176.198) added. New cstate: current_term: 1 leader_uuid: "b4abc88ed2e54830ae3a317629916117" committed_config { opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:28.475224 13603 consensus_queue.cc:237] T 8477215e8d4043c581c4ce7e5aa0fbd6 P 8446b7a7ea67486288d174eea7a3b3c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 42, Committed index: 42, Last appended: 1.42, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 43 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "7505fefeaf804ed8aeb3672f386ab959" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 38645 } attrs { promote: true } }
I20240502 01:53:28.477653 13734 raft_consensus.cc:1261] T 8477215e8d4043c581c4ce7e5aa0fbd6 P b4abc88ed2e54830ae3a317629916117 [term 1 FOLLOWER]: Refusing update from remote peer 8446b7a7ea67486288d174eea7a3b3c6: Log matching property violated. Preceding OpId in replica: term: 1 index: 42. Preceding OpId from leader: term: 1 index: 43. (index mismatch)
I20240502 01:53:28.481840 15449 consensus_queue.cc:1035] T 8477215e8d4043c581c4ce7e5aa0fbd6 P 8446b7a7ea67486288d174eea7a3b3c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 43, Last known committed idx: 42, Time since last communication: 0.000s
I20240502 01:53:28.485823 15449 raft_consensus.cc:2932] T 8477215e8d4043c581c4ce7e5aa0fbd6 P 8446b7a7ea67486288d174eea7a3b3c6 [term 1 LEADER]: Committing config change with OpId 1.43: config changed from index -1 to 43, NON_VOTER 7505fefeaf804ed8aeb3672f386ab959 (127.9.176.199) added. New config: { opid_index: 43 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "7505fefeaf804ed8aeb3672f386ab959" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 38645 } attrs { promote: true } } }
I20240502 01:53:28.486990 13733 raft_consensus.cc:2932] T 8477215e8d4043c581c4ce7e5aa0fbd6 P b4abc88ed2e54830ae3a317629916117 [term 1 FOLLOWER]: Committing config change with OpId 1.43: config changed from index -1 to 43, NON_VOTER 7505fefeaf804ed8aeb3672f386ab959 (127.9.176.199) added. New config: { opid_index: 43 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "7505fefeaf804ed8aeb3672f386ab959" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 38645 } attrs { promote: true } } }
I20240502 01:53:28.493567 13318 catalog_manager.cc:5484] T 8477215e8d4043c581c4ce7e5aa0fbd6 P 8446b7a7ea67486288d174eea7a3b3c6 reported cstate change: config changed from index -1 to 43, NON_VOTER 7505fefeaf804ed8aeb3672f386ab959 (127.9.176.199) added. New cstate: current_term: 1 leader_uuid: "8446b7a7ea67486288d174eea7a3b3c6" committed_config { opid_index: 43 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7505fefeaf804ed8aeb3672f386ab959" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 38645 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:28.580829 13733 consensus_queue.cc:237] T edf0aa94bfb94d499015897713ed94fa P b4abc88ed2e54830ae3a317629916117 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 54, Committed index: 54, Last appended: 1.54, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 55 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } attrs { replace: true } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } } peers { permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 41329 } attrs { promote: true } }
I20240502 01:53:28.600265 13470 raft_consensus.cc:1261] T edf0aa94bfb94d499015897713ed94fa P 5359629f948641f4b56e35ad6b6a15c4 [term 1 FOLLOWER]: Refusing update from remote peer b4abc88ed2e54830ae3a317629916117: Log matching property violated. Preceding OpId in replica: term: 1 index: 54. Preceding OpId from leader: term: 1 index: 55. (index mismatch)
I20240502 01:53:28.604255 15504 consensus_queue.cc:1035] T edf0aa94bfb94d499015897713ed94fa P b4abc88ed2e54830ae3a317629916117 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 55, Last known committed idx: 54, Time since last communication: 0.000s
I20240502 01:53:28.613705 15369 raft_consensus.cc:2932] T edf0aa94bfb94d499015897713ed94fa P b4abc88ed2e54830ae3a317629916117 [term 1 LEADER]: Committing config change with OpId 1.55: config changed from index -1 to 55, NON_VOTER ce90a4bd06ba41dd8e5aa227de9780a5 (127.9.176.196) added. New config: { opid_index: 55 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } attrs { replace: true } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } } peers { permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 41329 } attrs { promote: true } } }
W20240502 01:53:28.621608 13539 consensus_peers.cc:485] T 8477215e8d4043c581c4ce7e5aa0fbd6 P 8446b7a7ea67486288d174eea7a3b3c6 -> Peer 7505fefeaf804ed8aeb3672f386ab959 (127.9.176.199:38645): Couldn't send request to peer 7505fefeaf804ed8aeb3672f386ab959. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8477215e8d4043c581c4ce7e5aa0fbd6. This is attempt 1: this message will repeat every 5th retry.
W20240502 01:53:28.625885 13668 consensus_peers.cc:485] T edf0aa94bfb94d499015897713ed94fa P b4abc88ed2e54830ae3a317629916117 -> Peer ce90a4bd06ba41dd8e5aa227de9780a5 (127.9.176.196:41329): Couldn't send request to peer ce90a4bd06ba41dd8e5aa227de9780a5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: edf0aa94bfb94d499015897713ed94fa. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:28.625587 13470 raft_consensus.cc:2932] T edf0aa94bfb94d499015897713ed94fa P 5359629f948641f4b56e35ad6b6a15c4 [term 1 FOLLOWER]: Committing config change with OpId 1.55: config changed from index -1 to 55, NON_VOTER ce90a4bd06ba41dd8e5aa227de9780a5 (127.9.176.196) added. New config: { opid_index: 55 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } attrs { replace: true } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } } peers { permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 41329 } attrs { promote: true } } }
I20240502 01:53:28.625859 13318 catalog_manager.cc:5484] T edf0aa94bfb94d499015897713ed94fa P b4abc88ed2e54830ae3a317629916117 reported cstate change: config changed from index -1 to 55, NON_VOTER ce90a4bd06ba41dd8e5aa227de9780a5 (127.9.176.196) added. New cstate: current_term: 1 leader_uuid: "b4abc88ed2e54830ae3a317629916117" committed_config { opid_index: 55 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 41329 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:28.682655 15538 ts_tablet_manager.cc:902] T 4c0ab7c2995d4ccfa170f365b4e09b1d P 7e74a3d5df6a483c95c9dabfc99577bc: Initiating tablet copy from peer b4abc88ed2e54830ae3a317629916117 (127.9.176.195:41211)
I20240502 01:53:28.697814 13733 consensus_queue.cc:237] T daed3123a96a496da83a6110e6943264 P b4abc88ed2e54830ae3a317629916117 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 69, Committed index: 69, Last appended: 1.69, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } attrs { replace: true } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } } peers { permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: true } }
I20240502 01:53:28.700950 13599 raft_consensus.cc:1261] T daed3123a96a496da83a6110e6943264 P 8446b7a7ea67486288d174eea7a3b3c6 [term 1 FOLLOWER]: Refusing update from remote peer b4abc88ed2e54830ae3a317629916117: Log matching property violated. Preceding OpId in replica: term: 1 index: 69. Preceding OpId from leader: term: 1 index: 70. (index mismatch)
W20240502 01:53:28.701586 13668 consensus_peers.cc:485] T daed3123a96a496da83a6110e6943264 P b4abc88ed2e54830ae3a317629916117 -> Peer 7e74a3d5df6a483c95c9dabfc99577bc (127.9.176.197:35757): Couldn't send request to peer 7e74a3d5df6a483c95c9dabfc99577bc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: daed3123a96a496da83a6110e6943264. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:28.701978 15404 consensus_queue.cc:1035] T daed3123a96a496da83a6110e6943264 P b4abc88ed2e54830ae3a317629916117 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 70, Last known committed idx: 69, Time since last communication: 0.000s
I20240502 01:53:28.709606 15369 raft_consensus.cc:2932] T daed3123a96a496da83a6110e6943264 P b4abc88ed2e54830ae3a317629916117 [term 1 LEADER]: Committing config change with OpId 1.70: config changed from index -1 to 70, NON_VOTER 7e74a3d5df6a483c95c9dabfc99577bc (127.9.176.197) added. New config: { opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } attrs { replace: true } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } } peers { permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: true } } }
I20240502 01:53:28.710040 13604 raft_consensus.cc:2932] T daed3123a96a496da83a6110e6943264 P 8446b7a7ea67486288d174eea7a3b3c6 [term 1 FOLLOWER]: Committing config change with OpId 1.70: config changed from index -1 to 70, NON_VOTER 7e74a3d5df6a483c95c9dabfc99577bc (127.9.176.197) added. New config: { opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } attrs { replace: true } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } } peers { permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: true } } }
I20240502 01:53:28.713037 15538 tablet_copy_client.cc:308] T 4c0ab7c2995d4ccfa170f365b4e09b1d P 7e74a3d5df6a483c95c9dabfc99577bc: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.195:41211
I20240502 01:53:28.715714 13318 catalog_manager.cc:5484] T daed3123a96a496da83a6110e6943264 P 8446b7a7ea67486288d174eea7a3b3c6 reported cstate change: config changed from index -1 to 70, NON_VOTER 7e74a3d5df6a483c95c9dabfc99577bc (127.9.176.197) added. New cstate: current_term: 1 leader_uuid: "b4abc88ed2e54830ae3a317629916117" committed_config { opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } attrs { replace: true } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } } peers { permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: true } } }
I20240502 01:53:28.858400 15551 ts_tablet_manager.cc:902] T 57765e8e3ff7438ca9e588db8e685a03 P 9e4fa889fab047baa5bd3dcd77b2ca8f: Initiating tablet copy from peer b4abc88ed2e54830ae3a317629916117 (127.9.176.195:41211)
I20240502 01:53:28.994877 13756 tablet_copy_service.cc:140] P b4abc88ed2e54830ae3a317629916117: Received BeginTabletCopySession request for tablet 4c0ab7c2995d4ccfa170f365b4e09b1d from peer 7e74a3d5df6a483c95c9dabfc99577bc ({username='slave'} at 127.9.176.197:37391)
I20240502 01:53:28.995224 13756 tablet_copy_service.cc:161] P b4abc88ed2e54830ae3a317629916117: Beginning new tablet copy session on tablet 4c0ab7c2995d4ccfa170f365b4e09b1d from peer 7e74a3d5df6a483c95c9dabfc99577bc at {username='slave'} at 127.9.176.197:37391: session id = 7e74a3d5df6a483c95c9dabfc99577bc-4c0ab7c2995d4ccfa170f365b4e09b1d
I20240502 01:53:28.999501 13756 tablet_copy_source_session.cc:205] T 4c0ab7c2995d4ccfa170f365b4e09b1d P b4abc88ed2e54830ae3a317629916117: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:29.006378 15538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c0ab7c2995d4ccfa170f365b4e09b1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:29.010461 15556 ts_tablet_manager.cc:902] T edf0aa94bfb94d499015897713ed94fa P ce90a4bd06ba41dd8e5aa227de9780a5: Initiating tablet copy from peer b4abc88ed2e54830ae3a317629916117 (127.9.176.195:41211)
I20240502 01:53:29.024893 15538 tablet_copy_client.cc:783] T 4c0ab7c2995d4ccfa170f365b4e09b1d P 7e74a3d5df6a483c95c9dabfc99577bc: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:29.025413 15538 tablet_copy_client.cc:647] T 4c0ab7c2995d4ccfa170f365b4e09b1d P 7e74a3d5df6a483c95c9dabfc99577bc: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:29.028761 15538 tablet_copy_client.cc:523] T 4c0ab7c2995d4ccfa170f365b4e09b1d P 7e74a3d5df6a483c95c9dabfc99577bc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:29.034111 15538 tablet_bootstrap.cc:492] T 4c0ab7c2995d4ccfa170f365b4e09b1d P 7e74a3d5df6a483c95c9dabfc99577bc: Bootstrap starting.
I20240502 01:53:29.043234 15556 tablet_copy_client.cc:308] T edf0aa94bfb94d499015897713ed94fa P ce90a4bd06ba41dd8e5aa227de9780a5: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.195:41211
I20240502 01:53:29.078797 13756 tablet_copy_service.cc:140] P b4abc88ed2e54830ae3a317629916117: Received BeginTabletCopySession request for tablet edf0aa94bfb94d499015897713ed94fa from peer ce90a4bd06ba41dd8e5aa227de9780a5 ({username='slave'} at 127.9.176.196:34241)
I20240502 01:53:29.079048 13756 tablet_copy_service.cc:161] P b4abc88ed2e54830ae3a317629916117: Beginning new tablet copy session on tablet edf0aa94bfb94d499015897713ed94fa from peer ce90a4bd06ba41dd8e5aa227de9780a5 at {username='slave'} at 127.9.176.196:34241: session id = ce90a4bd06ba41dd8e5aa227de9780a5-edf0aa94bfb94d499015897713ed94fa
I20240502 01:53:29.082106 13756 tablet_copy_source_session.cc:205] T edf0aa94bfb94d499015897713ed94fa P b4abc88ed2e54830ae3a317629916117: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:29.086342 15556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edf0aa94bfb94d499015897713ed94fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:29.086808 15538 log.cc:824] T 4c0ab7c2995d4ccfa170f365b4e09b1d P 7e74a3d5df6a483c95c9dabfc99577bc: Log is configured to *not* fsync() on all Append() calls
I20240502 01:53:29.088955 15562 ts_tablet_manager.cc:902] T 8477215e8d4043c581c4ce7e5aa0fbd6 P 7505fefeaf804ed8aeb3672f386ab959: Initiating tablet copy from peer 8446b7a7ea67486288d174eea7a3b3c6 (127.9.176.194:34851)
I20240502 01:53:29.091673 13471 consensus_queue.cc:237] T eb1cf83294e044b49e115323de67ed42 P 5359629f948641f4b56e35ad6b6a15c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 69, Committed index: 69, Last appended: 1.69, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: true } }
I20240502 01:53:29.096021 13734 raft_consensus.cc:1261] T eb1cf83294e044b49e115323de67ed42 P b4abc88ed2e54830ae3a317629916117 [term 1 FOLLOWER]: Refusing update from remote peer 5359629f948641f4b56e35ad6b6a15c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 69. Preceding OpId from leader: term: 1 index: 70. (index mismatch)
I20240502 01:53:29.096823 15360 consensus_queue.cc:1035] T eb1cf83294e044b49e115323de67ed42 P 5359629f948641f4b56e35ad6b6a15c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 70, Last known committed idx: 69, Time since last communication: 0.000s
I20240502 01:53:29.099592 15556 tablet_copy_client.cc:783] T edf0aa94bfb94d499015897713ed94fa P ce90a4bd06ba41dd8e5aa227de9780a5: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:29.100059 15556 tablet_copy_client.cc:647] T edf0aa94bfb94d499015897713ed94fa P ce90a4bd06ba41dd8e5aa227de9780a5: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:29.100028 15360 raft_consensus.cc:2932] T eb1cf83294e044b49e115323de67ed42 P 5359629f948641f4b56e35ad6b6a15c4 [term 1 LEADER]: Committing config change with OpId 1.70: config changed from index -1 to 70, NON_VOTER 9e4fa889fab047baa5bd3dcd77b2ca8f (127.9.176.198) added. New config: { opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: true } } }
I20240502 01:53:29.103655 15556 tablet_copy_client.cc:523] T edf0aa94bfb94d499015897713ed94fa P ce90a4bd06ba41dd8e5aa227de9780a5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:29.105304 13734 raft_consensus.cc:2932] T eb1cf83294e044b49e115323de67ed42 P b4abc88ed2e54830ae3a317629916117 [term 1 FOLLOWER]: Committing config change with OpId 1.70: config changed from index -1 to 70, NON_VOTER 9e4fa889fab047baa5bd3dcd77b2ca8f (127.9.176.198) added. New config: { opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: true } } }
I20240502 01:53:29.111763 15556 tablet_bootstrap.cc:492] T edf0aa94bfb94d499015897713ed94fa P ce90a4bd06ba41dd8e5aa227de9780a5: Bootstrap starting.
I20240502 01:53:29.118862 15562 tablet_copy_client.cc:308] T 8477215e8d4043c581c4ce7e5aa0fbd6 P 7505fefeaf804ed8aeb3672f386ab959: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.194:34851
I20240502 01:53:29.128701 13323 catalog_manager.cc:5484] T eb1cf83294e044b49e115323de67ed42 P 5359629f948641f4b56e35ad6b6a15c4 reported cstate change: config changed from index -1 to 70, NON_VOTER 9e4fa889fab047baa5bd3dcd77b2ca8f (127.9.176.198) added. New cstate: current_term: 1 leader_uuid: "5359629f948641f4b56e35ad6b6a15c4" committed_config { opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:29.198398 15551 tablet_copy_client.cc:308] T 57765e8e3ff7438ca9e588db8e685a03 P 9e4fa889fab047baa5bd3dcd77b2ca8f: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.195:41211
I20240502 01:53:29.224179 13756 tablet_copy_service.cc:140] P b4abc88ed2e54830ae3a317629916117: Received BeginTabletCopySession request for tablet 57765e8e3ff7438ca9e588db8e685a03 from peer 9e4fa889fab047baa5bd3dcd77b2ca8f ({username='slave'} at 127.9.176.198:37121)
I20240502 01:53:29.224401 13756 tablet_copy_service.cc:161] P b4abc88ed2e54830ae3a317629916117: Beginning new tablet copy session on tablet 57765e8e3ff7438ca9e588db8e685a03 from peer 9e4fa889fab047baa5bd3dcd77b2ca8f at {username='slave'} at 127.9.176.198:37121: session id = 9e4fa889fab047baa5bd3dcd77b2ca8f-57765e8e3ff7438ca9e588db8e685a03
I20240502 01:53:29.228760 13756 tablet_copy_source_session.cc:205] T 57765e8e3ff7438ca9e588db8e685a03 P b4abc88ed2e54830ae3a317629916117: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:29.231402 15538 tablet_bootstrap.cc:492] T 4c0ab7c2995d4ccfa170f365b4e09b1d P 7e74a3d5df6a483c95c9dabfc99577bc: Bootstrap replayed 1/1 log segments. Stats: ops{read=67 overwritten=0 applied=67 ignored=0} inserts{seen=698 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:53:29.232296 15538 tablet_bootstrap.cc:492] T 4c0ab7c2995d4ccfa170f365b4e09b1d P 7e74a3d5df6a483c95c9dabfc99577bc: Bootstrap complete.
I20240502 01:53:29.232949 15538 ts_tablet_manager.cc:1367] T 4c0ab7c2995d4ccfa170f365b4e09b1d P 7e74a3d5df6a483c95c9dabfc99577bc: Time spent bootstrapping tablet: real 0.199s	user 0.136s	sys 0.012s
I20240502 01:53:29.235026 15551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57765e8e3ff7438ca9e588db8e685a03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:29.240945 15538 raft_consensus.cc:348] T 4c0ab7c2995d4ccfa170f365b4e09b1d P 7e74a3d5df6a483c95c9dabfc99577bc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 53 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } attrs { replace: true } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } } peers { permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: true } }
I20240502 01:53:29.241711 15538 raft_consensus.cc:729] T 4c0ab7c2995d4ccfa170f365b4e09b1d P 7e74a3d5df6a483c95c9dabfc99577bc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7e74a3d5df6a483c95c9dabfc99577bc, State: Initialized, Role: LEARNER
I20240502 01:53:29.242446 15538 consensus_queue.cc:260] T 4c0ab7c2995d4ccfa170f365b4e09b1d P 7e74a3d5df6a483c95c9dabfc99577bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 67, Last appended: 1.67, Last appended by leader: 67, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 53 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } attrs { replace: true } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } } peers { permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: true } }
I20240502 01:53:29.251138 15538 ts_tablet_manager.cc:1398] T 4c0ab7c2995d4ccfa170f365b4e09b1d P 7e74a3d5df6a483c95c9dabfc99577bc: Time spent starting tablet: real 0.018s	user 0.011s	sys 0.004s
I20240502 01:53:29.252398 13756 tablet_copy_service.cc:342] P b4abc88ed2e54830ae3a317629916117: Request end of tablet copy session 7e74a3d5df6a483c95c9dabfc99577bc-4c0ab7c2995d4ccfa170f365b4e09b1d received from {username='slave'} at 127.9.176.197:37391
I20240502 01:53:29.252763 13756 tablet_copy_service.cc:433] P b4abc88ed2e54830ae3a317629916117: ending tablet copy session 7e74a3d5df6a483c95c9dabfc99577bc-4c0ab7c2995d4ccfa170f365b4e09b1d on tablet 4c0ab7c2995d4ccfa170f365b4e09b1d with peer 7e74a3d5df6a483c95c9dabfc99577bc
I20240502 01:53:29.267494 13624 tablet_copy_service.cc:140] P 8446b7a7ea67486288d174eea7a3b3c6: Received BeginTabletCopySession request for tablet 8477215e8d4043c581c4ce7e5aa0fbd6 from peer 7505fefeaf804ed8aeb3672f386ab959 ({username='slave'} at 127.9.176.199:37323)
I20240502 01:53:29.267830 13624 tablet_copy_service.cc:161] P 8446b7a7ea67486288d174eea7a3b3c6: Beginning new tablet copy session on tablet 8477215e8d4043c581c4ce7e5aa0fbd6 from peer 7505fefeaf804ed8aeb3672f386ab959 at {username='slave'} at 127.9.176.199:37323: session id = 7505fefeaf804ed8aeb3672f386ab959-8477215e8d4043c581c4ce7e5aa0fbd6
I20240502 01:53:29.270478 15551 tablet_copy_client.cc:783] T 57765e8e3ff7438ca9e588db8e685a03 P 9e4fa889fab047baa5bd3dcd77b2ca8f: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:29.270882 15551 tablet_copy_client.cc:647] T 57765e8e3ff7438ca9e588db8e685a03 P 9e4fa889fab047baa5bd3dcd77b2ca8f: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:29.274214 13624 tablet_copy_source_session.cc:205] T 8477215e8d4043c581c4ce7e5aa0fbd6 P 8446b7a7ea67486288d174eea7a3b3c6: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:29.274924 15551 tablet_copy_client.cc:523] T 57765e8e3ff7438ca9e588db8e685a03 P 9e4fa889fab047baa5bd3dcd77b2ca8f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:29.276234 15538 ts_tablet_manager.cc:902] T daed3123a96a496da83a6110e6943264 P 7e74a3d5df6a483c95c9dabfc99577bc: Initiating tablet copy from peer b4abc88ed2e54830ae3a317629916117 (127.9.176.195:41211)
I20240502 01:53:29.279263 15551 tablet_bootstrap.cc:492] T 57765e8e3ff7438ca9e588db8e685a03 P 9e4fa889fab047baa5bd3dcd77b2ca8f: Bootstrap starting.
W20240502 01:53:29.281946 13405 consensus_peers.cc:485] T eb1cf83294e044b49e115323de67ed42 P 5359629f948641f4b56e35ad6b6a15c4 -> Peer 9e4fa889fab047baa5bd3dcd77b2ca8f (127.9.176.198:38527): Couldn't send request to peer 9e4fa889fab047baa5bd3dcd77b2ca8f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: eb1cf83294e044b49e115323de67ed42. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:29.286427 15562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8477215e8d4043c581c4ce7e5aa0fbd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:29.354005 15562 tablet_copy_client.cc:783] T 8477215e8d4043c581c4ce7e5aa0fbd6 P 7505fefeaf804ed8aeb3672f386ab959: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:29.354704 15562 tablet_copy_client.cc:647] T 8477215e8d4043c581c4ce7e5aa0fbd6 P 7505fefeaf804ed8aeb3672f386ab959: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:29.354983 15551 log.cc:824] T 57765e8e3ff7438ca9e588db8e685a03 P 9e4fa889fab047baa5bd3dcd77b2ca8f: Log is configured to *not* fsync() on all Append() calls
I20240502 01:53:29.358660 15562 tablet_copy_client.cc:523] T 8477215e8d4043c581c4ce7e5aa0fbd6 P 7505fefeaf804ed8aeb3672f386ab959: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:29.370858 15562 tablet_bootstrap.cc:492] T 8477215e8d4043c581c4ce7e5aa0fbd6 P 7505fefeaf804ed8aeb3672f386ab959: Bootstrap starting.
I20240502 01:53:29.462698 15562 log.cc:824] T 8477215e8d4043c581c4ce7e5aa0fbd6 P 7505fefeaf804ed8aeb3672f386ab959: Log is configured to *not* fsync() on all Append() calls
I20240502 01:53:29.471011 15556 tablet_bootstrap.cc:492] T edf0aa94bfb94d499015897713ed94fa P ce90a4bd06ba41dd8e5aa227de9780a5: Bootstrap replayed 1/1 log segments. Stats: ops{read=65 overwritten=0 applied=65 ignored=0} inserts{seen=750 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:53:29.471902 15556 tablet_bootstrap.cc:492] T edf0aa94bfb94d499015897713ed94fa P ce90a4bd06ba41dd8e5aa227de9780a5: Bootstrap complete.
I20240502 01:53:29.472635 15556 ts_tablet_manager.cc:1367] T edf0aa94bfb94d499015897713ed94fa P ce90a4bd06ba41dd8e5aa227de9780a5: Time spent bootstrapping tablet: real 0.361s	user 0.155s	sys 0.010s
I20240502 01:53:29.474622 15556 raft_consensus.cc:348] T edf0aa94bfb94d499015897713ed94fa P ce90a4bd06ba41dd8e5aa227de9780a5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 55 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } attrs { replace: true } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } } peers { permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 41329 } attrs { promote: true } }
I20240502 01:53:29.477346 15556 raft_consensus.cc:729] T edf0aa94bfb94d499015897713ed94fa P ce90a4bd06ba41dd8e5aa227de9780a5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ce90a4bd06ba41dd8e5aa227de9780a5, State: Initialized, Role: LEARNER
I20240502 01:53:29.478088 15556 consensus_queue.cc:260] T edf0aa94bfb94d499015897713ed94fa P ce90a4bd06ba41dd8e5aa227de9780a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 65, Last appended: 1.65, Last appended by leader: 65, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 55 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } attrs { replace: true } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } } peers { permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 41329 } attrs { promote: true } }
I20240502 01:53:29.497634 15590 ts_tablet_manager.cc:902] T eb1cf83294e044b49e115323de67ed42 P 9e4fa889fab047baa5bd3dcd77b2ca8f: Initiating tablet copy from peer 5359629f948641f4b56e35ad6b6a15c4 (127.9.176.193:33849)
I20240502 01:53:29.498920 15556 ts_tablet_manager.cc:1398] T edf0aa94bfb94d499015897713ed94fa P ce90a4bd06ba41dd8e5aa227de9780a5: Time spent starting tablet: real 0.026s	user 0.004s	sys 0.002s
I20240502 01:53:29.501278 14954 raft_consensus.cc:1203] T 4c0ab7c2995d4ccfa170f365b4e09b1d P 7e74a3d5df6a483c95c9dabfc99577bc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.52->[1.53-1.68]   Dedup: 1.67->[1.68-1.68]
I20240502 01:53:29.504269 15590 tablet_copy_client.cc:308] T eb1cf83294e044b49e115323de67ed42 P 9e4fa889fab047baa5bd3dcd77b2ca8f: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.193:33849
I20240502 01:53:29.510365 13756 tablet_copy_service.cc:342] P b4abc88ed2e54830ae3a317629916117: Request end of tablet copy session ce90a4bd06ba41dd8e5aa227de9780a5-edf0aa94bfb94d499015897713ed94fa received from {username='slave'} at 127.9.176.196:34241
I20240502 01:53:29.510548 13756 tablet_copy_service.cc:433] P b4abc88ed2e54830ae3a317629916117: ending tablet copy session ce90a4bd06ba41dd8e5aa227de9780a5-edf0aa94bfb94d499015897713ed94fa on tablet edf0aa94bfb94d499015897713ed94fa with peer ce90a4bd06ba41dd8e5aa227de9780a5
I20240502 01:53:29.514443 13491 tablet_copy_service.cc:140] P 5359629f948641f4b56e35ad6b6a15c4: Received BeginTabletCopySession request for tablet eb1cf83294e044b49e115323de67ed42 from peer 9e4fa889fab047baa5bd3dcd77b2ca8f ({username='slave'} at 127.9.176.198:54295)
I20240502 01:53:29.514766 13491 tablet_copy_service.cc:161] P 5359629f948641f4b56e35ad6b6a15c4: Beginning new tablet copy session on tablet eb1cf83294e044b49e115323de67ed42 from peer 9e4fa889fab047baa5bd3dcd77b2ca8f at {username='slave'} at 127.9.176.198:54295: session id = 9e4fa889fab047baa5bd3dcd77b2ca8f-eb1cf83294e044b49e115323de67ed42
I20240502 01:53:29.518870 13491 tablet_copy_source_session.cc:205] T eb1cf83294e044b49e115323de67ed42 P 5359629f948641f4b56e35ad6b6a15c4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:29.522465 15538 tablet_copy_client.cc:308] T daed3123a96a496da83a6110e6943264 P 7e74a3d5df6a483c95c9dabfc99577bc: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.195:41211
I20240502 01:53:29.522634 15590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb1cf83294e044b49e115323de67ed42. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:29.524678 13756 tablet_copy_service.cc:140] P b4abc88ed2e54830ae3a317629916117: Received BeginTabletCopySession request for tablet daed3123a96a496da83a6110e6943264 from peer 7e74a3d5df6a483c95c9dabfc99577bc ({username='slave'} at 127.9.176.197:37391)
I20240502 01:53:29.524904 13756 tablet_copy_service.cc:161] P b4abc88ed2e54830ae3a317629916117: Beginning new tablet copy session on tablet daed3123a96a496da83a6110e6943264 from peer 7e74a3d5df6a483c95c9dabfc99577bc at {username='slave'} at 127.9.176.197:37391: session id = 7e74a3d5df6a483c95c9dabfc99577bc-daed3123a96a496da83a6110e6943264
I20240502 01:53:29.527850 13756 tablet_copy_source_session.cc:205] T daed3123a96a496da83a6110e6943264 P b4abc88ed2e54830ae3a317629916117: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:29.529489 15538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet daed3123a96a496da83a6110e6943264. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:29.530448 15590 tablet_copy_client.cc:783] T eb1cf83294e044b49e115323de67ed42 P 9e4fa889fab047baa5bd3dcd77b2ca8f: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:29.530725 15590 tablet_copy_client.cc:647] T eb1cf83294e044b49e115323de67ed42 P 9e4fa889fab047baa5bd3dcd77b2ca8f: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:29.537827 15590 tablet_copy_client.cc:523] T eb1cf83294e044b49e115323de67ed42 P 9e4fa889fab047baa5bd3dcd77b2ca8f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:29.541450 15590 tablet_bootstrap.cc:492] T eb1cf83294e044b49e115323de67ed42 P 9e4fa889fab047baa5bd3dcd77b2ca8f: Bootstrap starting.
I20240502 01:53:29.546422 15538 tablet_copy_client.cc:783] T daed3123a96a496da83a6110e6943264 P 7e74a3d5df6a483c95c9dabfc99577bc: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:29.546747 15538 tablet_copy_client.cc:647] T daed3123a96a496da83a6110e6943264 P 7e74a3d5df6a483c95c9dabfc99577bc: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:29.556226 15538 tablet_copy_client.cc:523] T daed3123a96a496da83a6110e6943264 P 7e74a3d5df6a483c95c9dabfc99577bc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:29.564977 15538 tablet_bootstrap.cc:492] T daed3123a96a496da83a6110e6943264 P 7e74a3d5df6a483c95c9dabfc99577bc: Bootstrap starting.
I20240502 01:53:29.601624 15562 tablet_bootstrap.cc:492] T 8477215e8d4043c581c4ce7e5aa0fbd6 P 7505fefeaf804ed8aeb3672f386ab959: Bootstrap replayed 1/1 log segments. Stats: ops{read=52 overwritten=0 applied=51 ignored=0} inserts{seen=682 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 01:53:29.602512 15562 tablet_bootstrap.cc:492] T 8477215e8d4043c581c4ce7e5aa0fbd6 P 7505fefeaf804ed8aeb3672f386ab959: Bootstrap complete.
I20240502 01:53:29.603098 15562 ts_tablet_manager.cc:1367] T 8477215e8d4043c581c4ce7e5aa0fbd6 P 7505fefeaf804ed8aeb3672f386ab959: Time spent bootstrapping tablet: real 0.233s	user 0.135s	sys 0.007s
I20240502 01:53:29.615415 15562 raft_consensus.cc:348] T 8477215e8d4043c581c4ce7e5aa0fbd6 P 7505fefeaf804ed8aeb3672f386ab959 [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 43 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "7505fefeaf804ed8aeb3672f386ab959" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 38645 } attrs { promote: true } }
I20240502 01:53:29.631773 13734 consensus_queue.cc:237] T f2af8f5d5e784e1988a957666844db6b P b4abc88ed2e54830ae3a317629916117 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 60, Committed index: 60, Last appended: 1.60, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 61 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } attrs { replace: true } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } } peers { permanent_uuid: "7505fefeaf804ed8aeb3672f386ab959" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 38645 } attrs { promote: true } }
I20240502 01:53:29.644390 13599 raft_consensus.cc:1261] T f2af8f5d5e784e1988a957666844db6b P 8446b7a7ea67486288d174eea7a3b3c6 [term 1 FOLLOWER]: Refusing update from remote peer b4abc88ed2e54830ae3a317629916117: Log matching property violated. Preceding OpId in replica: term: 1 index: 60. Preceding OpId from leader: term: 1 index: 61. (index mismatch)
I20240502 01:53:29.645175 15369 consensus_queue.cc:1035] T f2af8f5d5e784e1988a957666844db6b P b4abc88ed2e54830ae3a317629916117 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 61, Last known committed idx: 60, Time since last communication: 0.000s
I20240502 01:53:29.648041 15562 raft_consensus.cc:729] T 8477215e8d4043c581c4ce7e5aa0fbd6 P 7505fefeaf804ed8aeb3672f386ab959 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7505fefeaf804ed8aeb3672f386ab959, State: Initialized, Role: LEARNER
I20240502 01:53:29.648404 15369 raft_consensus.cc:2932] T f2af8f5d5e784e1988a957666844db6b P b4abc88ed2e54830ae3a317629916117 [term 1 LEADER]: Committing config change with OpId 1.61: config changed from index -1 to 61, NON_VOTER 7505fefeaf804ed8aeb3672f386ab959 (127.9.176.199) added. New config: { opid_index: 61 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } attrs { replace: true } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } } peers { permanent_uuid: "7505fefeaf804ed8aeb3672f386ab959" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 38645 } attrs { promote: true } } }
I20240502 01:53:29.649224 15562 consensus_queue.cc:260] T 8477215e8d4043c581c4ce7e5aa0fbd6 P 7505fefeaf804ed8aeb3672f386ab959 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 51, Last appended: 1.52, Last appended by leader: 52, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 43 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "7505fefeaf804ed8aeb3672f386ab959" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 38645 } attrs { promote: true } }
I20240502 01:53:29.652566 13604 raft_consensus.cc:2932] T f2af8f5d5e784e1988a957666844db6b P 8446b7a7ea67486288d174eea7a3b3c6 [term 1 FOLLOWER]: Committing config change with OpId 1.61: config changed from index -1 to 61, NON_VOTER 7505fefeaf804ed8aeb3672f386ab959 (127.9.176.199) added. New config: { opid_index: 61 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } attrs { replace: true } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } } peers { permanent_uuid: "7505fefeaf804ed8aeb3672f386ab959" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 38645 } attrs { promote: true } } }
I20240502 01:53:29.654853 15562 ts_tablet_manager.cc:1398] T 8477215e8d4043c581c4ce7e5aa0fbd6 P 7505fefeaf804ed8aeb3672f386ab959: Time spent starting tablet: real 0.052s	user 0.010s	sys 0.012s
I20240502 01:53:29.656471 13624 tablet_copy_service.cc:342] P 8446b7a7ea67486288d174eea7a3b3c6: Request end of tablet copy session 7505fefeaf804ed8aeb3672f386ab959-8477215e8d4043c581c4ce7e5aa0fbd6 received from {username='slave'} at 127.9.176.199:37323
I20240502 01:53:29.656690 13624 tablet_copy_service.cc:433] P 8446b7a7ea67486288d174eea7a3b3c6: ending tablet copy session 7505fefeaf804ed8aeb3672f386ab959-8477215e8d4043c581c4ce7e5aa0fbd6 on tablet 8477215e8d4043c581c4ce7e5aa0fbd6 with peer 7505fefeaf804ed8aeb3672f386ab959
I20240502 01:53:29.664265 13323 catalog_manager.cc:5484] T f2af8f5d5e784e1988a957666844db6b P 8446b7a7ea67486288d174eea7a3b3c6 reported cstate change: config changed from index -1 to 61, NON_VOTER 7505fefeaf804ed8aeb3672f386ab959 (127.9.176.199) added. New cstate: current_term: 1 leader_uuid: "b4abc88ed2e54830ae3a317629916117" committed_config { opid_index: 61 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } attrs { replace: true } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } } peers { permanent_uuid: "7505fefeaf804ed8aeb3672f386ab959" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 38645 } attrs { promote: true } } }
I20240502 01:53:29.696879 15228 raft_consensus.cc:1203] T 8477215e8d4043c581c4ce7e5aa0fbd6 P 7505fefeaf804ed8aeb3672f386ab959 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.42->[1.43-1.52]   Dedup: 1.52->[]
I20240502 01:53:29.701035 15551 tablet_bootstrap.cc:492] T 57765e8e3ff7438ca9e588db8e685a03 P 9e4fa889fab047baa5bd3dcd77b2ca8f: Bootstrap replayed 1/1 log segments. Stats: ops{read=75 overwritten=0 applied=74 ignored=0} inserts{seen=759 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 01:53:29.701969 15551 tablet_bootstrap.cc:492] T 57765e8e3ff7438ca9e588db8e685a03 P 9e4fa889fab047baa5bd3dcd77b2ca8f: Bootstrap complete.
I20240502 01:53:29.702718 15551 ts_tablet_manager.cc:1367] T 57765e8e3ff7438ca9e588db8e685a03 P 9e4fa889fab047baa5bd3dcd77b2ca8f: Time spent bootstrapping tablet: real 0.424s	user 0.170s	sys 0.022s
I20240502 01:53:29.713320 15551 raft_consensus.cc:348] T 57765e8e3ff7438ca9e588db8e685a03 P 9e4fa889fab047baa5bd3dcd77b2ca8f [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } attrs { replace: true } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: true } }
I20240502 01:53:29.724400 15551 raft_consensus.cc:729] T 57765e8e3ff7438ca9e588db8e685a03 P 9e4fa889fab047baa5bd3dcd77b2ca8f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9e4fa889fab047baa5bd3dcd77b2ca8f, State: Initialized, Role: LEARNER
I20240502 01:53:29.725602 15551 consensus_queue.cc:260] T 57765e8e3ff7438ca9e588db8e685a03 P 9e4fa889fab047baa5bd3dcd77b2ca8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 74, Last appended: 1.75, Last appended by leader: 75, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } attrs { replace: true } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: true } }
I20240502 01:53:29.734560 15551 ts_tablet_manager.cc:1398] T 57765e8e3ff7438ca9e588db8e685a03 P 9e4fa889fab047baa5bd3dcd77b2ca8f: Time spent starting tablet: real 0.032s	user 0.020s	sys 0.008s
I20240502 01:53:29.739298 13756 tablet_copy_service.cc:342] P b4abc88ed2e54830ae3a317629916117: Request end of tablet copy session 9e4fa889fab047baa5bd3dcd77b2ca8f-57765e8e3ff7438ca9e588db8e685a03 received from {username='slave'} at 127.9.176.198:37121
I20240502 01:53:29.739457 13756 tablet_copy_service.cc:433] P b4abc88ed2e54830ae3a317629916117: ending tablet copy session 9e4fa889fab047baa5bd3dcd77b2ca8f-57765e8e3ff7438ca9e588db8e685a03 on tablet 57765e8e3ff7438ca9e588db8e685a03 with peer 9e4fa889fab047baa5bd3dcd77b2ca8f
I20240502 01:53:29.770169 15087 raft_consensus.cc:1203] T 57765e8e3ff7438ca9e588db8e685a03 P 9e4fa889fab047baa5bd3dcd77b2ca8f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.65->[1.66-1.78]   Dedup: 1.75->[1.76-1.78]
I20240502 01:53:29.834981 13471 consensus_queue.cc:237] T bdf6095f84aa497ba5db95d55c010085 P 5359629f948641f4b56e35ad6b6a15c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 70, Committed index: 70, Last appended: 1.71, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } attrs { replace: true } } peers { permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 41329 } attrs { promote: true } }
I20240502 01:53:29.882881 13604 raft_consensus.cc:1261] T bdf6095f84aa497ba5db95d55c010085 P 8446b7a7ea67486288d174eea7a3b3c6 [term 1 FOLLOWER]: Refusing update from remote peer 5359629f948641f4b56e35ad6b6a15c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 71. Preceding OpId from leader: term: 1 index: 72. (index mismatch)
I20240502 01:53:29.896664 15405 consensus_queue.cc:1035] T bdf6095f84aa497ba5db95d55c010085 P 5359629f948641f4b56e35ad6b6a15c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 72, Last known committed idx: 70, Time since last communication: 0.000s
I20240502 01:53:29.900411 15404 raft_consensus.cc:1050] T 4c0ab7c2995d4ccfa170f365b4e09b1d P b4abc88ed2e54830ae3a317629916117: attempting to promote NON_VOTER 7e74a3d5df6a483c95c9dabfc99577bc to VOTER
I20240502 01:53:29.908825 15404 consensus_queue.cc:237] T 4c0ab7c2995d4ccfa170f365b4e09b1d P b4abc88ed2e54830ae3a317629916117 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 68, Committed index: 68, Last appended: 1.69, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } attrs { replace: true } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } } peers { permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: false } }
I20240502 01:53:29.912966 14954 raft_consensus.cc:1261] T 4c0ab7c2995d4ccfa170f365b4e09b1d P 7e74a3d5df6a483c95c9dabfc99577bc [term 1 LEARNER]: Refusing update from remote peer b4abc88ed2e54830ae3a317629916117: Log matching property violated. Preceding OpId in replica: term: 1 index: 69. Preceding OpId from leader: term: 1 index: 70. (index mismatch)
I20240502 01:53:29.914907 15369 consensus_queue.cc:1035] T 4c0ab7c2995d4ccfa170f365b4e09b1d P b4abc88ed2e54830ae3a317629916117 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 70, Last known committed idx: 68, Time since last communication: 0.000s
I20240502 01:53:29.915282 15590 tablet_bootstrap.cc:492] T eb1cf83294e044b49e115323de67ed42 P 9e4fa889fab047baa5bd3dcd77b2ca8f: Bootstrap replayed 1/1 log segments. Stats: ops{read=70 overwritten=0 applied=70 ignored=0} inserts{seen=755 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:53:29.915915 15590 tablet_bootstrap.cc:492] T eb1cf83294e044b49e115323de67ed42 P 9e4fa889fab047baa5bd3dcd77b2ca8f: Bootstrap complete.
I20240502 01:53:29.916308 15590 ts_tablet_manager.cc:1367] T eb1cf83294e044b49e115323de67ed42 P 9e4fa889fab047baa5bd3dcd77b2ca8f: Time spent bootstrapping tablet: real 0.375s	user 0.135s	sys 0.018s
I20240502 01:53:29.917448 15590 raft_consensus.cc:348] T eb1cf83294e044b49e115323de67ed42 P 9e4fa889fab047baa5bd3dcd77b2ca8f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: true } }
I20240502 01:53:29.917793 15590 raft_consensus.cc:729] T eb1cf83294e044b49e115323de67ed42 P 9e4fa889fab047baa5bd3dcd77b2ca8f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9e4fa889fab047baa5bd3dcd77b2ca8f, State: Initialized, Role: LEARNER
I20240502 01:53:29.918069 15590 consensus_queue.cc:260] T eb1cf83294e044b49e115323de67ed42 P 9e4fa889fab047baa5bd3dcd77b2ca8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 70, Last appended: 1.70, Last appended by leader: 70, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: true } }
I20240502 01:53:29.918851 15590 ts_tablet_manager.cc:1398] T eb1cf83294e044b49e115323de67ed42 P 9e4fa889fab047baa5bd3dcd77b2ca8f: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240502 01:53:29.919935 13491 tablet_copy_service.cc:342] P 5359629f948641f4b56e35ad6b6a15c4: Request end of tablet copy session 9e4fa889fab047baa5bd3dcd77b2ca8f-eb1cf83294e044b49e115323de67ed42 received from {username='slave'} at 127.9.176.198:54295
I20240502 01:53:29.920166 13491 tablet_copy_service.cc:433] P 5359629f948641f4b56e35ad6b6a15c4: ending tablet copy session 9e4fa889fab047baa5bd3dcd77b2ca8f-eb1cf83294e044b49e115323de67ed42 on tablet eb1cf83294e044b49e115323de67ed42 with peer 9e4fa889fab047baa5bd3dcd77b2ca8f
I20240502 01:53:29.925074 14954 raft_consensus.cc:2932] T 4c0ab7c2995d4ccfa170f365b4e09b1d P 7e74a3d5df6a483c95c9dabfc99577bc [term 1 FOLLOWER]: Committing config change with OpId 1.70: config changed from index 53 to 70, 7e74a3d5df6a483c95c9dabfc99577bc (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } attrs { replace: true } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } } peers { permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: false } } }
W20240502 01:53:29.933933 13671 consensus_peers.cc:485] T f2af8f5d5e784e1988a957666844db6b P b4abc88ed2e54830ae3a317629916117 -> Peer 7505fefeaf804ed8aeb3672f386ab959 (127.9.176.199:38645): Couldn't send request to peer 7505fefeaf804ed8aeb3672f386ab959. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f2af8f5d5e784e1988a957666844db6b. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:29.933568 15504 raft_consensus.cc:2932] T 4c0ab7c2995d4ccfa170f365b4e09b1d P b4abc88ed2e54830ae3a317629916117 [term 1 LEADER]: Committing config change with OpId 1.70: config changed from index 53 to 70, 7e74a3d5df6a483c95c9dabfc99577bc (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } attrs { replace: true } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } } peers { permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: false } } }
I20240502 01:53:29.934746 13470 raft_consensus.cc:1261] T 4c0ab7c2995d4ccfa170f365b4e09b1d P 5359629f948641f4b56e35ad6b6a15c4 [term 1 FOLLOWER]: Refusing update from remote peer b4abc88ed2e54830ae3a317629916117: Log matching property violated. Preceding OpId in replica: term: 1 index: 69. Preceding OpId from leader: term: 1 index: 70. (index mismatch)
I20240502 01:53:29.935983 15404 consensus_queue.cc:1035] T 4c0ab7c2995d4ccfa170f365b4e09b1d P b4abc88ed2e54830ae3a317629916117 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 70, Last known committed idx: 69, Time since last communication: 0.000s
I20240502 01:53:29.938136 13470 raft_consensus.cc:2932] T 4c0ab7c2995d4ccfa170f365b4e09b1d P 5359629f948641f4b56e35ad6b6a15c4 [term 1 FOLLOWER]: Committing config change with OpId 1.70: config changed from index 53 to 70, 7e74a3d5df6a483c95c9dabfc99577bc (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } attrs { replace: true } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } } peers { permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: false } } }
I20240502 01:53:29.944566 15087 raft_consensus.cc:1203] T eb1cf83294e044b49e115323de67ed42 P 9e4fa889fab047baa5bd3dcd77b2ca8f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.69->[1.70-1.73]   Dedup: 1.70->[1.71-1.73]
I20240502 01:53:29.946753 13318 catalog_manager.cc:5484] T 4c0ab7c2995d4ccfa170f365b4e09b1d P 5359629f948641f4b56e35ad6b6a15c4 reported cstate change: config changed from index 53 to 70, 7e74a3d5df6a483c95c9dabfc99577bc (127.9.176.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b4abc88ed2e54830ae3a317629916117" committed_config { opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } attrs { replace: true } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } } peers { permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: false } } }
I20240502 01:53:29.971990 13733 consensus_queue.cc:237] T 4c0ab7c2995d4ccfa170f365b4e09b1d P b4abc88ed2e54830ae3a317629916117 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 72, Committed index: 72, Last appended: 1.72, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 73 OBSOLETE_local: false peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } } peers { permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: false } }
I20240502 01:53:29.977807 14953 raft_consensus.cc:1261] T 4c0ab7c2995d4ccfa170f365b4e09b1d P 7e74a3d5df6a483c95c9dabfc99577bc [term 1 FOLLOWER]: Refusing update from remote peer b4abc88ed2e54830ae3a317629916117: Log matching property violated. Preceding OpId in replica: term: 1 index: 71. Preceding OpId from leader: term: 1 index: 73. (index mismatch)
I20240502 01:53:29.979053 15624 consensus_queue.cc:1035] T 4c0ab7c2995d4ccfa170f365b4e09b1d P b4abc88ed2e54830ae3a317629916117 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 73, Last known committed idx: 70, Time since last communication: 0.000s
I20240502 01:53:29.985304 15404 raft_consensus.cc:2932] T 4c0ab7c2995d4ccfa170f365b4e09b1d P b4abc88ed2e54830ae3a317629916117 [term 1 LEADER]: Committing config change with OpId 1.73: config changed from index 70 to 73, VOTER 5359629f948641f4b56e35ad6b6a15c4 (127.9.176.193) evicted. New config: { opid_index: 73 OBSOLETE_local: false peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } } peers { permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: false } } }
I20240502 01:53:29.989264 15356 raft_consensus.cc:2932] T bdf6095f84aa497ba5db95d55c010085 P 5359629f948641f4b56e35ad6b6a15c4 [term 1 LEADER]: Committing config change with OpId 1.72: config changed from index -1 to 72, NON_VOTER ce90a4bd06ba41dd8e5aa227de9780a5 (127.9.176.196) added. New config: { opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } attrs { replace: true } } peers { permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 41329 } attrs { promote: true } } }
I20240502 01:53:29.990449 14953 raft_consensus.cc:2932] T 4c0ab7c2995d4ccfa170f365b4e09b1d P 7e74a3d5df6a483c95c9dabfc99577bc [term 1 FOLLOWER]: Committing config change with OpId 1.73: config changed from index 70 to 73, VOTER 5359629f948641f4b56e35ad6b6a15c4 (127.9.176.193) evicted. New config: { opid_index: 73 OBSOLETE_local: false peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } } peers { permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: false } } }
I20240502 01:53:29.996476 13326 catalog_manager.cc:5484] T 4c0ab7c2995d4ccfa170f365b4e09b1d P 7e74a3d5df6a483c95c9dabfc99577bc reported cstate change: config changed from index 70 to 73, VOTER 5359629f948641f4b56e35ad6b6a15c4 (127.9.176.193) evicted. New cstate: current_term: 1 leader_uuid: "b4abc88ed2e54830ae3a317629916117" committed_config { opid_index: 73 OBSOLETE_local: false peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } } peers { permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: false } } }
I20240502 01:53:29.999279 13312 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4c0ab7c2995d4ccfa170f365b4e09b1d with cas_config_opid_index 70: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:53:30.013859 13323 catalog_manager.cc:5484] T bdf6095f84aa497ba5db95d55c010085 P 5359629f948641f4b56e35ad6b6a15c4 reported cstate change: config changed from index -1 to 72, NON_VOTER ce90a4bd06ba41dd8e5aa227de9780a5 (127.9.176.196) added. New cstate: current_term: 1 leader_uuid: "5359629f948641f4b56e35ad6b6a15c4" committed_config { opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 41329 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:30.025430 13604 raft_consensus.cc:2932] T bdf6095f84aa497ba5db95d55c010085 P 8446b7a7ea67486288d174eea7a3b3c6 [term 1 FOLLOWER]: Committing config change with OpId 1.72: config changed from index -1 to 72, NON_VOTER ce90a4bd06ba41dd8e5aa227de9780a5 (127.9.176.196) added. New config: { opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } attrs { replace: true } } peers { permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 41329 } attrs { promote: true } } }
I20240502 01:53:30.046993 15476 raft_consensus.cc:1050] T eb1cf83294e044b49e115323de67ed42 P 5359629f948641f4b56e35ad6b6a15c4: attempting to promote NON_VOTER 9e4fa889fab047baa5bd3dcd77b2ca8f to VOTER
I20240502 01:53:30.048462 15476 consensus_queue.cc:237] T eb1cf83294e044b49e115323de67ed42 P 5359629f948641f4b56e35ad6b6a15c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 74, Committed index: 74, Last appended: 1.74, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 75 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false } }
I20240502 01:53:30.051102 13733 raft_consensus.cc:1261] T eb1cf83294e044b49e115323de67ed42 P b4abc88ed2e54830ae3a317629916117 [term 1 FOLLOWER]: Refusing update from remote peer 5359629f948641f4b56e35ad6b6a15c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 74. Preceding OpId from leader: term: 1 index: 75. (index mismatch)
I20240502 01:53:30.051304 15086 raft_consensus.cc:1261] T eb1cf83294e044b49e115323de67ed42 P 9e4fa889fab047baa5bd3dcd77b2ca8f [term 1 LEARNER]: Refusing update from remote peer 5359629f948641f4b56e35ad6b6a15c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 74. Preceding OpId from leader: term: 1 index: 75. (index mismatch)
I20240502 01:53:30.052191 15405 consensus_queue.cc:1035] T eb1cf83294e044b49e115323de67ed42 P 5359629f948641f4b56e35ad6b6a15c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 75, Last known committed idx: 74, Time since last communication: 0.000s
I20240502 01:53:30.053810 14821 raft_consensus.cc:1203] T edf0aa94bfb94d499015897713ed94fa P ce90a4bd06ba41dd8e5aa227de9780a5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.54->[1.55-1.66]   Dedup: 1.65->[1.66-1.66]
I20240502 01:53:30.057070 15405 consensus_queue.cc:1035] T eb1cf83294e044b49e115323de67ed42 P 5359629f948641f4b56e35ad6b6a15c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 75, Last known committed idx: 74, Time since last communication: 0.000s
I20240502 01:53:30.061158 15360 raft_consensus.cc:2932] T eb1cf83294e044b49e115323de67ed42 P 5359629f948641f4b56e35ad6b6a15c4 [term 1 LEADER]: Committing config change with OpId 1.75: config changed from index 70 to 75, 9e4fa889fab047baa5bd3dcd77b2ca8f (127.9.176.198) changed from NON_VOTER to VOTER. New config: { opid_index: 75 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false } } }
I20240502 01:53:30.062789 14820 consensus_queue.cc:237] T d38a05f7ffa94cc5abc50a76a83692b5 P ce90a4bd06ba41dd8e5aa227de9780a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } attrs { replace: true } } peers { permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 41329 } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: true } }
I20240502 01:53:30.065444 13732 raft_consensus.cc:2932] T eb1cf83294e044b49e115323de67ed42 P b4abc88ed2e54830ae3a317629916117 [term 1 FOLLOWER]: Committing config change with OpId 1.75: config changed from index 70 to 75, 9e4fa889fab047baa5bd3dcd77b2ca8f (127.9.176.198) changed from NON_VOTER to VOTER. New config: { opid_index: 75 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false } } }
I20240502 01:53:30.073866 13318 catalog_manager.cc:5484] T eb1cf83294e044b49e115323de67ed42 P b4abc88ed2e54830ae3a317629916117 reported cstate change: config changed from index 70 to 75, 9e4fa889fab047baa5bd3dcd77b2ca8f (127.9.176.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5359629f948641f4b56e35ad6b6a15c4" committed_config { opid_index: 75 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false } } }
I20240502 01:53:30.078109 15086 raft_consensus.cc:2932] T eb1cf83294e044b49e115323de67ed42 P 9e4fa889fab047baa5bd3dcd77b2ca8f [term 1 FOLLOWER]: Committing config change with OpId 1.75: config changed from index 70 to 75, 9e4fa889fab047baa5bd3dcd77b2ca8f (127.9.176.198) changed from NON_VOTER to VOTER. New config: { opid_index: 75 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false } } }
I20240502 01:53:30.079921 13451 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c0ab7c2995d4ccfa170f365b4e09b1d with delete_type TABLET_DATA_TOMBSTONED (TS 5359629f948641f4b56e35ad6b6a15c4 not found in new config with opid_index 73) from {username='slave'} at 127.0.0.1:38182
W20240502 01:53:30.081771 13402 consensus_peers.cc:485] T bdf6095f84aa497ba5db95d55c010085 P 5359629f948641f4b56e35ad6b6a15c4 -> Peer ce90a4bd06ba41dd8e5aa227de9780a5 (127.9.176.196:41329): Couldn't send request to peer ce90a4bd06ba41dd8e5aa227de9780a5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bdf6095f84aa497ba5db95d55c010085. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:30.142263 15449 raft_consensus.cc:1050] T 8477215e8d4043c581c4ce7e5aa0fbd6 P 8446b7a7ea67486288d174eea7a3b3c6: attempting to promote NON_VOTER 7505fefeaf804ed8aeb3672f386ab959 to VOTER
I20240502 01:53:30.143286 15449 consensus_queue.cc:237] T 8477215e8d4043c581c4ce7e5aa0fbd6 P 8446b7a7ea67486288d174eea7a3b3c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 52, Committed index: 52, Last appended: 1.53, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 54 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "7505fefeaf804ed8aeb3672f386ab959" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 38645 } attrs { promote: false } }
I20240502 01:53:30.145756 15228 raft_consensus.cc:1261] T 8477215e8d4043c581c4ce7e5aa0fbd6 P 7505fefeaf804ed8aeb3672f386ab959 [term 1 LEARNER]: Refusing update from remote peer 8446b7a7ea67486288d174eea7a3b3c6: Log matching property violated. Preceding OpId in replica: term: 1 index: 53. Preceding OpId from leader: term: 1 index: 54. (index mismatch)
I20240502 01:53:30.146802 15498 consensus_queue.cc:1035] T 8477215e8d4043c581c4ce7e5aa0fbd6 P 8446b7a7ea67486288d174eea7a3b3c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7505fefeaf804ed8aeb3672f386ab959" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 38645 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 54, Last known committed idx: 52, Time since last communication: 0.000s
I20240502 01:53:30.155376 15228 raft_consensus.cc:2932] T 8477215e8d4043c581c4ce7e5aa0fbd6 P 7505fefeaf804ed8aeb3672f386ab959 [term 1 FOLLOWER]: Committing config change with OpId 1.54: config changed from index 43 to 54, 7505fefeaf804ed8aeb3672f386ab959 (127.9.176.199) changed from NON_VOTER to VOTER. New config: { opid_index: 54 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "7505fefeaf804ed8aeb3672f386ab959" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 38645 } attrs { promote: false } } }
I20240502 01:53:30.167917 13735 raft_consensus.cc:1261] T 8477215e8d4043c581c4ce7e5aa0fbd6 P b4abc88ed2e54830ae3a317629916117 [term 1 FOLLOWER]: Refusing update from remote peer 8446b7a7ea67486288d174eea7a3b3c6: Log matching property violated. Preceding OpId in replica: term: 1 index: 53. Preceding OpId from leader: term: 1 index: 54. (index mismatch)
I20240502 01:53:30.169008 15538 tablet_bootstrap.cc:492] T daed3123a96a496da83a6110e6943264 P 7e74a3d5df6a483c95c9dabfc99577bc: Bootstrap replayed 1/1 log segments. Stats: ops{read=75 overwritten=0 applied=75 ignored=0} inserts{seen=760 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:53:30.169469 15538 tablet_bootstrap.cc:492] T daed3123a96a496da83a6110e6943264 P 7e74a3d5df6a483c95c9dabfc99577bc: Bootstrap complete.
I20240502 01:53:30.169782 15538 ts_tablet_manager.cc:1367] T daed3123a96a496da83a6110e6943264 P 7e74a3d5df6a483c95c9dabfc99577bc: Time spent bootstrapping tablet: real 0.605s	user 0.267s	sys 0.148s
I20240502 01:53:30.170926 15449 consensus_queue.cc:1035] T 8477215e8d4043c581c4ce7e5aa0fbd6 P 8446b7a7ea67486288d174eea7a3b3c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 54, Last known committed idx: 52, Time since last communication: 0.000s
I20240502 01:53:30.170905 15538 raft_consensus.cc:348] T daed3123a96a496da83a6110e6943264 P 7e74a3d5df6a483c95c9dabfc99577bc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } attrs { replace: true } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } } peers { permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: true } }
I20240502 01:53:30.171299 15538 raft_consensus.cc:729] T daed3123a96a496da83a6110e6943264 P 7e74a3d5df6a483c95c9dabfc99577bc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7e74a3d5df6a483c95c9dabfc99577bc, State: Initialized, Role: LEARNER
I20240502 01:53:30.171551 15538 consensus_queue.cc:260] T daed3123a96a496da83a6110e6943264 P 7e74a3d5df6a483c95c9dabfc99577bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 75, Last appended: 1.75, Last appended by leader: 75, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } attrs { replace: true } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } } peers { permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: true } }
I20240502 01:53:30.172588 15538 ts_tablet_manager.cc:1398] T daed3123a96a496da83a6110e6943264 P 7e74a3d5df6a483c95c9dabfc99577bc: Time spent starting tablet: real 0.003s	user 0.004s	sys 0.000s
I20240502 01:53:30.173619 13756 tablet_copy_service.cc:342] P b4abc88ed2e54830ae3a317629916117: Request end of tablet copy session 7e74a3d5df6a483c95c9dabfc99577bc-daed3123a96a496da83a6110e6943264 received from {username='slave'} at 127.9.176.197:37391
I20240502 01:53:30.173795 13756 tablet_copy_service.cc:433] P b4abc88ed2e54830ae3a317629916117: ending tablet copy session 7e74a3d5df6a483c95c9dabfc99577bc-daed3123a96a496da83a6110e6943264 on tablet daed3123a96a496da83a6110e6943264 with peer 7e74a3d5df6a483c95c9dabfc99577bc
I20240502 01:53:30.178591 13735 raft_consensus.cc:2932] T 8477215e8d4043c581c4ce7e5aa0fbd6 P b4abc88ed2e54830ae3a317629916117 [term 1 FOLLOWER]: Committing config change with OpId 1.54: config changed from index 43 to 54, 7505fefeaf804ed8aeb3672f386ab959 (127.9.176.199) changed from NON_VOTER to VOTER. New config: { opid_index: 54 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "7505fefeaf804ed8aeb3672f386ab959" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 38645 } attrs { promote: false } } }
I20240502 01:53:30.235968 15652 ts_tablet_manager.cc:902] T f2af8f5d5e784e1988a957666844db6b P 7505fefeaf804ed8aeb3672f386ab959: Initiating tablet copy from peer b4abc88ed2e54830ae3a317629916117 (127.9.176.195:41211)
I20240502 01:53:30.253871 15498 raft_consensus.cc:2932] T 8477215e8d4043c581c4ce7e5aa0fbd6 P 8446b7a7ea67486288d174eea7a3b3c6 [term 1 LEADER]: Committing config change with OpId 1.54: config changed from index 43 to 54, 7505fefeaf804ed8aeb3672f386ab959 (127.9.176.199) changed from NON_VOTER to VOTER. New config: { opid_index: 54 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "7505fefeaf804ed8aeb3672f386ab959" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 38645 } attrs { promote: false } } }
I20240502 01:53:30.257702 13471 raft_consensus.cc:1261] T d38a05f7ffa94cc5abc50a76a83692b5 P 5359629f948641f4b56e35ad6b6a15c4 [term 2 FOLLOWER]: Refusing update from remote peer ce90a4bd06ba41dd8e5aa227de9780a5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 01:53:30.258687 15638 tablet_replica.cc:330] T 4c0ab7c2995d4ccfa170f365b4e09b1d P 5359629f948641f4b56e35ad6b6a15c4: stopping tablet replica
I20240502 01:53:30.259348 15638 raft_consensus.cc:2229] T 4c0ab7c2995d4ccfa170f365b4e09b1d P 5359629f948641f4b56e35ad6b6a15c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:30.260031 15638 raft_consensus.cc:2258] T 4c0ab7c2995d4ccfa170f365b4e09b1d P 5359629f948641f4b56e35ad6b6a15c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:30.268200 15638 ts_tablet_manager.cc:1875] T 4c0ab7c2995d4ccfa170f365b4e09b1d P 5359629f948641f4b56e35ad6b6a15c4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:53:30.272650 13317 catalog_manager.cc:5484] T 8477215e8d4043c581c4ce7e5aa0fbd6 P 7505fefeaf804ed8aeb3672f386ab959 reported cstate change: config changed from index 43 to 54, 7505fefeaf804ed8aeb3672f386ab959 (127.9.176.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8446b7a7ea67486288d174eea7a3b3c6" committed_config { opid_index: 54 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "7505fefeaf804ed8aeb3672f386ab959" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 38645 } attrs { promote: false } } }
I20240502 01:53:30.273986 15637 consensus_queue.cc:1035] T d38a05f7ffa94cc5abc50a76a83692b5 P ce90a4bd06ba41dd8e5aa227de9780a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:53:30.282711 13470 consensus_queue.cc:237] T eb1cf83294e044b49e115323de67ed42 P 5359629f948641f4b56e35ad6b6a15c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 76, Committed index: 76, Last appended: 1.76, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false } }
I20240502 01:53:30.289618 13602 consensus_queue.cc:237] T 8477215e8d4043c581c4ce7e5aa0fbd6 P 8446b7a7ea67486288d174eea7a3b3c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 55, Committed index: 55, Last appended: 1.55, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 56 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } } peers { permanent_uuid: "7505fefeaf804ed8aeb3672f386ab959" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 38645 } attrs { promote: false } }
I20240502 01:53:30.295739 15504 raft_consensus.cc:1050] T edf0aa94bfb94d499015897713ed94fa P b4abc88ed2e54830ae3a317629916117: attempting to promote NON_VOTER ce90a4bd06ba41dd8e5aa227de9780a5 to VOTER
I20240502 01:53:30.295748 15227 raft_consensus.cc:1261] T 8477215e8d4043c581c4ce7e5aa0fbd6 P 7505fefeaf804ed8aeb3672f386ab959 [term 1 FOLLOWER]: Refusing update from remote peer 8446b7a7ea67486288d174eea7a3b3c6: Log matching property violated. Preceding OpId in replica: term: 1 index: 55. Preceding OpId from leader: term: 1 index: 56. (index mismatch)
I20240502 01:53:30.296409 15498 consensus_queue.cc:1035] T 8477215e8d4043c581c4ce7e5aa0fbd6 P 8446b7a7ea67486288d174eea7a3b3c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7505fefeaf804ed8aeb3672f386ab959" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 38645 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 56, Last known committed idx: 55, Time since last communication: 0.000s
I20240502 01:53:30.296671 15504 consensus_queue.cc:237] T edf0aa94bfb94d499015897713ed94fa P b4abc88ed2e54830ae3a317629916117 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 69, Committed index: 69, Last appended: 1.69, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } attrs { replace: true } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } } peers { permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 41329 } attrs { promote: false } }
I20240502 01:53:30.299472 13470 raft_consensus.cc:1261] T edf0aa94bfb94d499015897713ed94fa P 5359629f948641f4b56e35ad6b6a15c4 [term 1 FOLLOWER]: Refusing update from remote peer b4abc88ed2e54830ae3a317629916117: Log matching property violated. Preceding OpId in replica: term: 1 index: 69. Preceding OpId from leader: term: 1 index: 70. (index mismatch)
I20240502 01:53:30.299993 14821 raft_consensus.cc:1261] T edf0aa94bfb94d499015897713ed94fa P ce90a4bd06ba41dd8e5aa227de9780a5 [term 1 LEARNER]: Refusing update from remote peer b4abc88ed2e54830ae3a317629916117: Log matching property violated. Preceding OpId in replica: term: 1 index: 69. Preceding OpId from leader: term: 1 index: 70. (index mismatch)
I20240502 01:53:30.299834 15449 raft_consensus.cc:2932] T 8477215e8d4043c581c4ce7e5aa0fbd6 P 8446b7a7ea67486288d174eea7a3b3c6 [term 1 LEADER]: Committing config change with OpId 1.56: config changed from index 54 to 56, VOTER b4abc88ed2e54830ae3a317629916117 (127.9.176.195) evicted. New config: { opid_index: 56 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } } peers { permanent_uuid: "7505fefeaf804ed8aeb3672f386ab959" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 38645 } attrs { promote: false } } }
I20240502 01:53:30.300272 15625 consensus_queue.cc:1035] T edf0aa94bfb94d499015897713ed94fa P b4abc88ed2e54830ae3a317629916117 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 70, Last known committed idx: 69, Time since last communication: 0.000s
I20240502 01:53:30.301410 15625 consensus_queue.cc:1035] T edf0aa94bfb94d499015897713ed94fa P b4abc88ed2e54830ae3a317629916117 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 41329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 70, Last known committed idx: 69, Time since last communication: 0.000s
I20240502 01:53:30.304919 15228 raft_consensus.cc:2932] T 8477215e8d4043c581c4ce7e5aa0fbd6 P 7505fefeaf804ed8aeb3672f386ab959 [term 1 FOLLOWER]: Committing config change with OpId 1.56: config changed from index 54 to 56, VOTER b4abc88ed2e54830ae3a317629916117 (127.9.176.195) evicted. New config: { opid_index: 56 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } } peers { permanent_uuid: "7505fefeaf804ed8aeb3672f386ab959" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 38645 } attrs { promote: false } } }
I20240502 01:53:30.304973 15625 raft_consensus.cc:2932] T edf0aa94bfb94d499015897713ed94fa P b4abc88ed2e54830ae3a317629916117 [term 1 LEADER]: Committing config change with OpId 1.70: config changed from index 55 to 70, ce90a4bd06ba41dd8e5aa227de9780a5 (127.9.176.196) changed from NON_VOTER to VOTER. New config: { opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } attrs { replace: true } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } } peers { permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 41329 } attrs { promote: false } } }
I20240502 01:53:30.310146 13310 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8477215e8d4043c581c4ce7e5aa0fbd6 with cas_config_opid_index 54: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:53:30.310961 14821 raft_consensus.cc:2932] T edf0aa94bfb94d499015897713ed94fa P ce90a4bd06ba41dd8e5aa227de9780a5 [term 1 FOLLOWER]: Committing config change with OpId 1.70: config changed from index 55 to 70, ce90a4bd06ba41dd8e5aa227de9780a5 (127.9.176.196) changed from NON_VOTER to VOTER. New config: { opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } attrs { replace: true } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } } peers { permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 41329 } attrs { promote: false } } }
I20240502 01:53:30.312358 13470 raft_consensus.cc:2932] T edf0aa94bfb94d499015897713ed94fa P 5359629f948641f4b56e35ad6b6a15c4 [term 1 FOLLOWER]: Committing config change with OpId 1.70: config changed from index 55 to 70, ce90a4bd06ba41dd8e5aa227de9780a5 (127.9.176.196) changed from NON_VOTER to VOTER. New config: { opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } attrs { replace: true } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } } peers { permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 41329 } attrs { promote: false } } }
I20240502 01:53:30.312721 13325 catalog_manager.cc:5484] T 8477215e8d4043c581c4ce7e5aa0fbd6 P 8446b7a7ea67486288d174eea7a3b3c6 reported cstate change: config changed from index 54 to 56, VOTER b4abc88ed2e54830ae3a317629916117 (127.9.176.195) evicted. New cstate: current_term: 1 leader_uuid: "8446b7a7ea67486288d174eea7a3b3c6" committed_config { opid_index: 56 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7505fefeaf804ed8aeb3672f386ab959" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 38645 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:53:30.317780 15087 raft_consensus.cc:1261] T eb1cf83294e044b49e115323de67ed42 P 9e4fa889fab047baa5bd3dcd77b2ca8f [term 1 FOLLOWER]: Refusing update from remote peer 5359629f948641f4b56e35ad6b6a15c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 76. Preceding OpId from leader: term: 1 index: 77. (index mismatch)
I20240502 01:53:30.319115 13318 catalog_manager.cc:5484] T edf0aa94bfb94d499015897713ed94fa P 5359629f948641f4b56e35ad6b6a15c4 reported cstate change: config changed from index 55 to 70, ce90a4bd06ba41dd8e5aa227de9780a5 (127.9.176.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b4abc88ed2e54830ae3a317629916117" committed_config { opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } attrs { replace: true } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } } peers { permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 41329 } attrs { promote: false } } }
I20240502 01:53:30.336719 13715 tablet_service.cc:1514] Processing DeleteTablet for tablet 8477215e8d4043c581c4ce7e5aa0fbd6 with delete_type TABLET_DATA_TOMBSTONED (TS b4abc88ed2e54830ae3a317629916117 not found in new config with opid_index 56) from {username='slave'} at 127.0.0.1:44986
I20240502 01:53:30.343528 15476 consensus_queue.cc:1035] T eb1cf83294e044b49e115323de67ed42 P 5359629f948641f4b56e35ad6b6a15c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 77, Last known committed idx: 76, Time since last communication: 0.000s
I20240502 01:53:30.352418 15504 raft_consensus.cc:1050] T 57765e8e3ff7438ca9e588db8e685a03 P b4abc88ed2e54830ae3a317629916117: attempting to promote NON_VOTER 9e4fa889fab047baa5bd3dcd77b2ca8f to VOTER
I20240502 01:53:30.353317 15504 consensus_queue.cc:237] T 57765e8e3ff7438ca9e588db8e685a03 P b4abc88ed2e54830ae3a317629916117 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 80, Committed index: 80, Last appended: 1.80, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } attrs { replace: true } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false } }
I20240502 01:53:30.356480 13735 consensus_queue.cc:237] T edf0aa94bfb94d499015897713ed94fa P b4abc88ed2e54830ae3a317629916117 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 71, Committed index: 71, Last appended: 1.71, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } } peers { permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 41329 } attrs { promote: false } }
I20240502 01:53:30.359968 15660 tablet_replica.cc:330] T 8477215e8d4043c581c4ce7e5aa0fbd6 P b4abc88ed2e54830ae3a317629916117: stopping tablet replica
I20240502 01:53:30.360316 14820 raft_consensus.cc:1261] T edf0aa94bfb94d499015897713ed94fa P ce90a4bd06ba41dd8e5aa227de9780a5 [term 1 FOLLOWER]: Refusing update from remote peer b4abc88ed2e54830ae3a317629916117: Log matching property violated. Preceding OpId in replica: term: 1 index: 71. Preceding OpId from leader: term: 1 index: 72. (index mismatch)
I20240502 01:53:30.360895 13602 raft_consensus.cc:1261] T 57765e8e3ff7438ca9e588db8e685a03 P 8446b7a7ea67486288d174eea7a3b3c6 [term 1 FOLLOWER]: Refusing update from remote peer b4abc88ed2e54830ae3a317629916117: Log matching property violated. Preceding OpId in replica: term: 1 index: 80. Preceding OpId from leader: term: 1 index: 81. (index mismatch)
W20240502 01:53:30.361452 14754 consensus_peers.cc:485] T d38a05f7ffa94cc5abc50a76a83692b5 P ce90a4bd06ba41dd8e5aa227de9780a5 -> Peer 9e4fa889fab047baa5bd3dcd77b2ca8f (127.9.176.198:38527): Couldn't send request to peer 9e4fa889fab047baa5bd3dcd77b2ca8f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d38a05f7ffa94cc5abc50a76a83692b5. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:30.363493 15360 raft_consensus.cc:2932] T eb1cf83294e044b49e115323de67ed42 P 5359629f948641f4b56e35ad6b6a15c4 [term 1 LEADER]: Committing config change with OpId 1.77: config changed from index 75 to 77, VOTER b4abc88ed2e54830ae3a317629916117 (127.9.176.195) evicted. New config: { opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false } } }
I20240502 01:53:30.367807 15660 raft_consensus.cc:2229] T 8477215e8d4043c581c4ce7e5aa0fbd6 P b4abc88ed2e54830ae3a317629916117 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:30.368569 15660 pending_rounds.cc:62] T 8477215e8d4043c581c4ce7e5aa0fbd6 P b4abc88ed2e54830ae3a317629916117: Trying to abort 1 pending ops.
I20240502 01:53:30.371901 15660 pending_rounds.cc:68] T 8477215e8d4043c581c4ce7e5aa0fbd6 P b4abc88ed2e54830ae3a317629916117: Aborting op as it isn't in flight: id { term: 1 index: 55 } timestamp: 7023062262718803968 op_type: WRITE_OP write_request { tablet_id: "8477215e8d4043c581c4ce7e5aa0fbd6" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000*\330\325\000w$\202\217\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""01a5c1a7d299b939" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023062262670565376 authz_token { token_data: "\010\300\346\313\261\006\"3\n\005slave\022*\n c4a6875e444a446dbf9576f2a5d024c0\020\001\030\001 \001(\001" signature: "<redacted>""p\363(\274\257\362\202\352x\003X\322\357\025b\334\033\r\340\\\365\332\242\272PC\240<\263\033Y\313\250\270\213\332\264\000i\020\226s\3369\271\016\351^\263\346\342\346\333\010*\006\312<\342M>\024\313\351" signing_key_seq_num: 0 } } request_id { client_id: "6e2ee35b168944c9bf107e65d2931efc" seq_no: 194 first_incomplete_seq_no: 194 attempt_no: 0 }
I20240502 01:53:30.372805 13313 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet eb1cf83294e044b49e115323de67ed42 with cas_config_opid_index 75: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:53:30.373755 15660 raft_consensus.cc:2258] T 8477215e8d4043c581c4ce7e5aa0fbd6 P b4abc88ed2e54830ae3a317629916117 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:30.376333 15087 raft_consensus.cc:1261] T 57765e8e3ff7438ca9e588db8e685a03 P 9e4fa889fab047baa5bd3dcd77b2ca8f [term 1 LEARNER]: Refusing update from remote peer b4abc88ed2e54830ae3a317629916117: Log matching property violated. Preceding OpId in replica: term: 1 index: 80. Preceding OpId from leader: term: 1 index: 81. (index mismatch)
I20240502 01:53:30.377772 15087 raft_consensus.cc:2932] T eb1cf83294e044b49e115323de67ed42 P 9e4fa889fab047baa5bd3dcd77b2ca8f [term 1 FOLLOWER]: Committing config change with OpId 1.77: config changed from index 75 to 77, VOTER b4abc88ed2e54830ae3a317629916117 (127.9.176.195) evicted. New config: { opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false } } }
I20240502 01:53:30.377992 13325 catalog_manager.cc:5484] T eb1cf83294e044b49e115323de67ed42 P 5359629f948641f4b56e35ad6b6a15c4 reported cstate change: config changed from index 75 to 77, VOTER b4abc88ed2e54830ae3a317629916117 (127.9.176.195) evicted. New cstate: current_term: 1 leader_uuid: "5359629f948641f4b56e35ad6b6a15c4" committed_config { opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:53:30.379354 15625 consensus_queue.cc:1035] T 57765e8e3ff7438ca9e588db8e685a03 P b4abc88ed2e54830ae3a317629916117 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 81, Last known committed idx: 80, Time since last communication: 0.000s
I20240502 01:53:30.381537 15660 ts_tablet_manager.cc:1875] T 8477215e8d4043c581c4ce7e5aa0fbd6 P b4abc88ed2e54830ae3a317629916117: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:53:30.385665 15637 raft_consensus.cc:2932] T d38a05f7ffa94cc5abc50a76a83692b5 P ce90a4bd06ba41dd8e5aa227de9780a5 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 9e4fa889fab047baa5bd3dcd77b2ca8f (127.9.176.198) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } attrs { replace: true } } peers { permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 41329 } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: true } } }
I20240502 01:53:30.389434 15504 consensus_queue.cc:1035] T edf0aa94bfb94d499015897713ed94fa P b4abc88ed2e54830ae3a317629916117 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 41329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 72, Last known committed idx: 71, Time since last communication: 0.000s
I20240502 01:53:30.390228 15504 consensus_queue.cc:1035] T 57765e8e3ff7438ca9e588db8e685a03 P b4abc88ed2e54830ae3a317629916117 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 81, Last known committed idx: 80, Time since last communication: 0.000s
I20240502 01:53:30.400643 15660 ts_tablet_manager.cc:1888] T 8477215e8d4043c581c4ce7e5aa0fbd6 P b4abc88ed2e54830ae3a317629916117: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.55
I20240502 01:53:30.404026 15660 log.cc:1196] T 8477215e8d4043c581c4ce7e5aa0fbd6 P b4abc88ed2e54830ae3a317629916117: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/wal/wals/8477215e8d4043c581c4ce7e5aa0fbd6
I20240502 01:53:30.403753 15625 raft_consensus.cc:2932] T 57765e8e3ff7438ca9e588db8e685a03 P b4abc88ed2e54830ae3a317629916117 [term 1 LEADER]: Committing config change with OpId 1.81: config changed from index 66 to 81, 9e4fa889fab047baa5bd3dcd77b2ca8f (127.9.176.198) changed from NON_VOTER to VOTER. New config: { opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } attrs { replace: true } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false } } }
I20240502 01:53:30.406744 15652 tablet_copy_client.cc:308] T f2af8f5d5e784e1988a957666844db6b P 7505fefeaf804ed8aeb3672f386ab959: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.195:41211
I20240502 01:53:30.410858 13471 raft_consensus.cc:2932] T d38a05f7ffa94cc5abc50a76a83692b5 P 5359629f948641f4b56e35ad6b6a15c4 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 9e4fa889fab047baa5bd3dcd77b2ca8f (127.9.176.198) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } attrs { replace: true } } peers { permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 41329 } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: true } } }
I20240502 01:53:30.414175 13326 catalog_manager.cc:5484] T d38a05f7ffa94cc5abc50a76a83692b5 P ce90a4bd06ba41dd8e5aa227de9780a5 reported cstate change: config changed from index -1 to 3, NON_VOTER 9e4fa889fab047baa5bd3dcd77b2ca8f (127.9.176.198) added. New cstate: current_term: 2 leader_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 41329 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:30.422719 14954 raft_consensus.cc:1203] T daed3123a96a496da83a6110e6943264 P 7e74a3d5df6a483c95c9dabfc99577bc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.69->[1.70-1.79]   Dedup: 1.75->[1.76-1.79]
I20240502 01:53:30.422899 13312 catalog_manager.cc:4830] TS b4abc88ed2e54830ae3a317629916117 (127.9.176.195:41211): tablet 8477215e8d4043c581c4ce7e5aa0fbd6 (table rebalance_test_table_2 [id=c4a6875e444a446dbf9576f2a5d024c0]) successfully deleted
I20240502 01:53:30.423717 15404 raft_consensus.cc:2932] T edf0aa94bfb94d499015897713ed94fa P b4abc88ed2e54830ae3a317629916117 [term 1 LEADER]: Committing config change with OpId 1.72: config changed from index 70 to 72, VOTER 5359629f948641f4b56e35ad6b6a15c4 (127.9.176.193) evicted. New config: { opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } } peers { permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 41329 } attrs { promote: false } } }
I20240502 01:53:30.425311 13715 tablet_service.cc:1514] Processing DeleteTablet for tablet eb1cf83294e044b49e115323de67ed42 with delete_type TABLET_DATA_TOMBSTONED (TS b4abc88ed2e54830ae3a317629916117 not found in new config with opid_index 77) from {username='slave'} at 127.0.0.1:44986
I20240502 01:53:30.426229 15660 tablet_replica.cc:330] T eb1cf83294e044b49e115323de67ed42 P b4abc88ed2e54830ae3a317629916117: stopping tablet replica
I20240502 01:53:30.426462 15660 raft_consensus.cc:2229] T eb1cf83294e044b49e115323de67ed42 P b4abc88ed2e54830ae3a317629916117 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:30.426756 15660 raft_consensus.cc:2258] T eb1cf83294e044b49e115323de67ed42 P b4abc88ed2e54830ae3a317629916117 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:30.428206 15660 ts_tablet_manager.cc:1875] T eb1cf83294e044b49e115323de67ed42 P b4abc88ed2e54830ae3a317629916117: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:53:30.429160 14820 raft_consensus.cc:2932] T edf0aa94bfb94d499015897713ed94fa P ce90a4bd06ba41dd8e5aa227de9780a5 [term 1 FOLLOWER]: Committing config change with OpId 1.72: config changed from index 70 to 72, VOTER 5359629f948641f4b56e35ad6b6a15c4 (127.9.176.193) evicted. New config: { opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } } peers { permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 41329 } attrs { promote: false } } }
I20240502 01:53:30.430595 13312 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet edf0aa94bfb94d499015897713ed94fa with cas_config_opid_index 70: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:53:30.430980 13599 raft_consensus.cc:2932] T 57765e8e3ff7438ca9e588db8e685a03 P 8446b7a7ea67486288d174eea7a3b3c6 [term 1 FOLLOWER]: Committing config change with OpId 1.81: config changed from index 66 to 81, 9e4fa889fab047baa5bd3dcd77b2ca8f (127.9.176.198) changed from NON_VOTER to VOTER. New config: { opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } attrs { replace: true } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false } } }
I20240502 01:53:30.434157 13326 catalog_manager.cc:5484] T edf0aa94bfb94d499015897713ed94fa P b4abc88ed2e54830ae3a317629916117 reported cstate change: config changed from index 70 to 72, VOTER 5359629f948641f4b56e35ad6b6a15c4 (127.9.176.193) evicted. New cstate: current_term: 1 leader_uuid: "b4abc88ed2e54830ae3a317629916117" committed_config { opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 41329 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:53:30.436810 15660 ts_tablet_manager.cc:1888] T eb1cf83294e044b49e115323de67ed42 P b4abc88ed2e54830ae3a317629916117: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.76
I20240502 01:53:30.439936 15660 log.cc:1196] T eb1cf83294e044b49e115323de67ed42 P b4abc88ed2e54830ae3a317629916117: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/wal/wals/eb1cf83294e044b49e115323de67ed42
I20240502 01:53:30.442981 13312 catalog_manager.cc:4830] TS b4abc88ed2e54830ae3a317629916117 (127.9.176.195:41211): tablet eb1cf83294e044b49e115323de67ed42 (table rebalance_test_table_3 [id=c824fe62874b40458b3bf6ce96e3242b]) successfully deleted
I20240502 01:53:30.443471 13451 tablet_service.cc:1514] Processing DeleteTablet for tablet edf0aa94bfb94d499015897713ed94fa with delete_type TABLET_DATA_TOMBSTONED (TS 5359629f948641f4b56e35ad6b6a15c4 not found in new config with opid_index 72) from {username='slave'} at 127.0.0.1:38182
I20240502 01:53:30.459131 13325 catalog_manager.cc:5484] T 57765e8e3ff7438ca9e588db8e685a03 P 8446b7a7ea67486288d174eea7a3b3c6 reported cstate change: config changed from index 66 to 81, 9e4fa889fab047baa5bd3dcd77b2ca8f (127.9.176.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b4abc88ed2e54830ae3a317629916117" committed_config { opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } attrs { replace: true } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false } } }
I20240502 01:53:30.471738 13756 tablet_copy_service.cc:140] P b4abc88ed2e54830ae3a317629916117: Received BeginTabletCopySession request for tablet f2af8f5d5e784e1988a957666844db6b from peer 7505fefeaf804ed8aeb3672f386ab959 ({username='slave'} at 127.9.176.199:55115)
I20240502 01:53:30.471956 13756 tablet_copy_service.cc:161] P b4abc88ed2e54830ae3a317629916117: Beginning new tablet copy session on tablet f2af8f5d5e784e1988a957666844db6b from peer 7505fefeaf804ed8aeb3672f386ab959 at {username='slave'} at 127.9.176.199:55115: session id = 7505fefeaf804ed8aeb3672f386ab959-f2af8f5d5e784e1988a957666844db6b
I20240502 01:53:30.475440 13756 tablet_copy_source_session.cc:205] T f2af8f5d5e784e1988a957666844db6b P b4abc88ed2e54830ae3a317629916117: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:30.482143 15652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2af8f5d5e784e1988a957666844db6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:30.484987 15087 raft_consensus.cc:2932] T 57765e8e3ff7438ca9e588db8e685a03 P 9e4fa889fab047baa5bd3dcd77b2ca8f [term 1 FOLLOWER]: Committing config change with OpId 1.81: config changed from index 66 to 81, 9e4fa889fab047baa5bd3dcd77b2ca8f (127.9.176.198) changed from NON_VOTER to VOTER. New config: { opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } attrs { replace: true } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false } } }
I20240502 01:53:30.498832 15638 ts_tablet_manager.cc:1888] T 4c0ab7c2995d4ccfa170f365b4e09b1d P 5359629f948641f4b56e35ad6b6a15c4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.72
I20240502 01:53:30.499995 15652 tablet_copy_client.cc:783] T f2af8f5d5e784e1988a957666844db6b P 7505fefeaf804ed8aeb3672f386ab959: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:30.500355 15652 tablet_copy_client.cc:647] T f2af8f5d5e784e1988a957666844db6b P 7505fefeaf804ed8aeb3672f386ab959: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:30.502830 15652 tablet_copy_client.cc:523] T f2af8f5d5e784e1988a957666844db6b P 7505fefeaf804ed8aeb3672f386ab959: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:30.507134 15652 tablet_bootstrap.cc:492] T f2af8f5d5e784e1988a957666844db6b P 7505fefeaf804ed8aeb3672f386ab959: Bootstrap starting.
I20240502 01:53:30.512256 13734 consensus_queue.cc:237] T 57765e8e3ff7438ca9e588db8e685a03 P b4abc88ed2e54830ae3a317629916117 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 82, Committed index: 82, Last appended: 1.82, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false } }
I20240502 01:53:30.519122 15087 raft_consensus.cc:1261] T 57765e8e3ff7438ca9e588db8e685a03 P 9e4fa889fab047baa5bd3dcd77b2ca8f [term 1 FOLLOWER]: Refusing update from remote peer b4abc88ed2e54830ae3a317629916117: Log matching property violated. Preceding OpId in replica: term: 1 index: 82. Preceding OpId from leader: term: 1 index: 83. (index mismatch)
I20240502 01:53:30.519819 15624 consensus_queue.cc:1035] T 57765e8e3ff7438ca9e588db8e685a03 P b4abc88ed2e54830ae3a317629916117 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 83, Last known committed idx: 82, Time since last communication: 0.000s
I20240502 01:53:30.522538 15668 raft_consensus.cc:2932] T 57765e8e3ff7438ca9e588db8e685a03 P b4abc88ed2e54830ae3a317629916117 [term 1 LEADER]: Committing config change with OpId 1.83: config changed from index 81 to 83, VOTER 8446b7a7ea67486288d174eea7a3b3c6 (127.9.176.194) evicted. New config: { opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false } } }
I20240502 01:53:30.523305 15086 raft_consensus.cc:2932] T 57765e8e3ff7438ca9e588db8e685a03 P 9e4fa889fab047baa5bd3dcd77b2ca8f [term 1 FOLLOWER]: Committing config change with OpId 1.83: config changed from index 81 to 83, VOTER 8446b7a7ea67486288d174eea7a3b3c6 (127.9.176.194) evicted. New config: { opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false } } }
I20240502 01:53:30.529517 13326 catalog_manager.cc:5484] T 57765e8e3ff7438ca9e588db8e685a03 P 9e4fa889fab047baa5bd3dcd77b2ca8f reported cstate change: config changed from index 81 to 83, VOTER 8446b7a7ea67486288d174eea7a3b3c6 (127.9.176.194) evicted. New cstate: current_term: 1 leader_uuid: "b4abc88ed2e54830ae3a317629916117" committed_config { opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false } } }
I20240502 01:53:30.537403 13312 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 57765e8e3ff7438ca9e588db8e685a03 with cas_config_opid_index 81: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:53:30.537665 13584 tablet_service.cc:1514] Processing DeleteTablet for tablet 57765e8e3ff7438ca9e588db8e685a03 with delete_type TABLET_DATA_TOMBSTONED (TS 8446b7a7ea67486288d174eea7a3b3c6 not found in new config with opid_index 83) from {username='slave'} at 127.0.0.1:41580
I20240502 01:53:30.546707 15667 ts_tablet_manager.cc:902] T bdf6095f84aa497ba5db95d55c010085 P ce90a4bd06ba41dd8e5aa227de9780a5: Initiating tablet copy from peer 5359629f948641f4b56e35ad6b6a15c4 (127.9.176.193:33849)
I20240502 01:53:30.549021 15668 raft_consensus.cc:1050] T daed3123a96a496da83a6110e6943264 P b4abc88ed2e54830ae3a317629916117: attempting to promote NON_VOTER 7e74a3d5df6a483c95c9dabfc99577bc to VOTER
I20240502 01:53:30.549865 15668 consensus_queue.cc:237] T daed3123a96a496da83a6110e6943264 P b4abc88ed2e54830ae3a317629916117 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 80, Committed index: 80, Last appended: 1.80, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } attrs { replace: true } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } } peers { permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: false } }
I20240502 01:53:30.558413 14953 raft_consensus.cc:1261] T daed3123a96a496da83a6110e6943264 P 7e74a3d5df6a483c95c9dabfc99577bc [term 1 LEARNER]: Refusing update from remote peer b4abc88ed2e54830ae3a317629916117: Log matching property violated. Preceding OpId in replica: term: 1 index: 80. Preceding OpId from leader: term: 1 index: 81. (index mismatch)
I20240502 01:53:30.560237 13602 raft_consensus.cc:1261] T daed3123a96a496da83a6110e6943264 P 8446b7a7ea67486288d174eea7a3b3c6 [term 1 FOLLOWER]: Refusing update from remote peer b4abc88ed2e54830ae3a317629916117: Log matching property violated. Preceding OpId in replica: term: 1 index: 80. Preceding OpId from leader: term: 1 index: 81. (index mismatch)
I20240502 01:53:30.564592 15638 log.cc:1196] T 4c0ab7c2995d4ccfa170f365b4e09b1d P 5359629f948641f4b56e35ad6b6a15c4: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/wal/wals/4c0ab7c2995d4ccfa170f365b4e09b1d
I20240502 01:53:30.564990 15504 consensus_queue.cc:1035] T daed3123a96a496da83a6110e6943264 P b4abc88ed2e54830ae3a317629916117 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 81, Last known committed idx: 80, Time since last communication: 0.000s
I20240502 01:53:30.565647 15504 consensus_queue.cc:1035] T daed3123a96a496da83a6110e6943264 P b4abc88ed2e54830ae3a317629916117 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 81, Last known committed idx: 79, Time since last communication: 0.000s
I20240502 01:53:30.573650 15638 tablet_replica.cc:330] T edf0aa94bfb94d499015897713ed94fa P 5359629f948641f4b56e35ad6b6a15c4: stopping tablet replica
I20240502 01:53:30.573745 13313 catalog_manager.cc:4830] TS 5359629f948641f4b56e35ad6b6a15c4 (127.9.176.193:33849): tablet 4c0ab7c2995d4ccfa170f365b4e09b1d (table rebalance_test_table_3 [id=c824fe62874b40458b3bf6ce96e3242b]) successfully deleted
I20240502 01:53:30.574151 15638 raft_consensus.cc:2229] T edf0aa94bfb94d499015897713ed94fa P 5359629f948641f4b56e35ad6b6a15c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:30.576653 15638 raft_consensus.cc:2258] T edf0aa94bfb94d499015897713ed94fa P 5359629f948641f4b56e35ad6b6a15c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:30.576828 15674 tablet_replica.cc:330] T 57765e8e3ff7438ca9e588db8e685a03 P 8446b7a7ea67486288d174eea7a3b3c6: stopping tablet replica
I20240502 01:53:30.577509 15674 raft_consensus.cc:2229] T 57765e8e3ff7438ca9e588db8e685a03 P 8446b7a7ea67486288d174eea7a3b3c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:30.578050 15674 raft_consensus.cc:2258] T 57765e8e3ff7438ca9e588db8e685a03 P 8446b7a7ea67486288d174eea7a3b3c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:30.578991 15404 raft_consensus.cc:2932] T daed3123a96a496da83a6110e6943264 P b4abc88ed2e54830ae3a317629916117 [term 1 LEADER]: Committing config change with OpId 1.81: config changed from index 70 to 81, 7e74a3d5df6a483c95c9dabfc99577bc (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } attrs { replace: true } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } } peers { permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: false } } }
I20240502 01:53:30.582851 15674 ts_tablet_manager.cc:1875] T 57765e8e3ff7438ca9e588db8e685a03 P 8446b7a7ea67486288d174eea7a3b3c6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:53:30.587091 13325 catalog_manager.cc:5484] T daed3123a96a496da83a6110e6943264 P b4abc88ed2e54830ae3a317629916117 reported cstate change: config changed from index 70 to 81, 7e74a3d5df6a483c95c9dabfc99577bc (127.9.176.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b4abc88ed2e54830ae3a317629916117" committed_config { opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:30.591857 13471 consensus_queue.cc:237] T b38a1fb4d1694c609a932e8816fc3514 P 5359629f948641f4b56e35ad6b6a15c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 63, Committed index: 63, Last appended: 1.63, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 64 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: true } }
I20240502 01:53:30.585932 13598 raft_consensus.cc:2932] T daed3123a96a496da83a6110e6943264 P 8446b7a7ea67486288d174eea7a3b3c6 [term 1 FOLLOWER]: Committing config change with OpId 1.81: config changed from index 70 to 81, 7e74a3d5df6a483c95c9dabfc99577bc (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } attrs { replace: true } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } } peers { permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: false } } }
I20240502 01:53:30.594280 13735 raft_consensus.cc:1261] T b38a1fb4d1694c609a932e8816fc3514 P b4abc88ed2e54830ae3a317629916117 [term 1 FOLLOWER]: Refusing update from remote peer 5359629f948641f4b56e35ad6b6a15c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 63. Preceding OpId from leader: term: 1 index: 64. (index mismatch)
I20240502 01:53:30.595954 15360 consensus_queue.cc:1035] T b38a1fb4d1694c609a932e8816fc3514 P 5359629f948641f4b56e35ad6b6a15c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 64, Last known committed idx: 63, Time since last communication: 0.000s
I20240502 01:53:30.599399 15360 raft_consensus.cc:2932] T b38a1fb4d1694c609a932e8816fc3514 P 5359629f948641f4b56e35ad6b6a15c4 [term 1 LEADER]: Committing config change with OpId 1.64: config changed from index -1 to 64, NON_VOTER 7e74a3d5df6a483c95c9dabfc99577bc (127.9.176.197) added. New config: { opid_index: 64 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: true } } }
I20240502 01:53:30.602056 13735 raft_consensus.cc:2932] T b38a1fb4d1694c609a932e8816fc3514 P b4abc88ed2e54830ae3a317629916117 [term 1 FOLLOWER]: Committing config change with OpId 1.64: config changed from index -1 to 64, NON_VOTER 7e74a3d5df6a483c95c9dabfc99577bc (127.9.176.197) added. New config: { opid_index: 64 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: true } } }
I20240502 01:53:30.617462 13326 catalog_manager.cc:5484] T b38a1fb4d1694c609a932e8816fc3514 P b4abc88ed2e54830ae3a317629916117 reported cstate change: config changed from index -1 to 64, NON_VOTER 7e74a3d5df6a483c95c9dabfc99577bc (127.9.176.197) added. New cstate: current_term: 1 leader_uuid: "5359629f948641f4b56e35ad6b6a15c4" committed_config { opid_index: 64 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: true } } }
I20240502 01:53:30.628886 14954 raft_consensus.cc:2932] T daed3123a96a496da83a6110e6943264 P 7e74a3d5df6a483c95c9dabfc99577bc [term 1 FOLLOWER]: Committing config change with OpId 1.81: config changed from index 70 to 81, 7e74a3d5df6a483c95c9dabfc99577bc (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } attrs { replace: true } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } } peers { permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: false } } }
I20240502 01:53:30.633227 15638 ts_tablet_manager.cc:1875] T edf0aa94bfb94d499015897713ed94fa P 5359629f948641f4b56e35ad6b6a15c4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:53:30.652456 13735 consensus_queue.cc:237] T daed3123a96a496da83a6110e6943264 P b4abc88ed2e54830ae3a317629916117 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 81, Committed index: 81, Last appended: 1.81, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } } peers { permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: false } }
I20240502 01:53:30.654889 15674 ts_tablet_manager.cc:1888] T 57765e8e3ff7438ca9e588db8e685a03 P 8446b7a7ea67486288d174eea7a3b3c6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.82
I20240502 01:53:30.659600 15667 tablet_copy_client.cc:308] T bdf6095f84aa497ba5db95d55c010085 P ce90a4bd06ba41dd8e5aa227de9780a5: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.193:33849
I20240502 01:53:30.660904 13491 tablet_copy_service.cc:140] P 5359629f948641f4b56e35ad6b6a15c4: Received BeginTabletCopySession request for tablet bdf6095f84aa497ba5db95d55c010085 from peer ce90a4bd06ba41dd8e5aa227de9780a5 ({username='slave'} at 127.9.176.196:54151)
I20240502 01:53:30.661202 13491 tablet_copy_service.cc:161] P 5359629f948641f4b56e35ad6b6a15c4: Beginning new tablet copy session on tablet bdf6095f84aa497ba5db95d55c010085 from peer ce90a4bd06ba41dd8e5aa227de9780a5 at {username='slave'} at 127.9.176.196:54151: session id = ce90a4bd06ba41dd8e5aa227de9780a5-bdf6095f84aa497ba5db95d55c010085
I20240502 01:53:30.665565 15674 log.cc:1196] T 57765e8e3ff7438ca9e588db8e685a03 P 8446b7a7ea67486288d174eea7a3b3c6: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/wal/wals/57765e8e3ff7438ca9e588db8e685a03
I20240502 01:53:30.670205 13491 tablet_copy_source_session.cc:205] T bdf6095f84aa497ba5db95d55c010085 P 5359629f948641f4b56e35ad6b6a15c4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:30.670650 13310 catalog_manager.cc:4830] TS 8446b7a7ea67486288d174eea7a3b3c6 (127.9.176.194:34851): tablet 57765e8e3ff7438ca9e588db8e685a03 (table rebalance_test_table_2 [id=c4a6875e444a446dbf9576f2a5d024c0]) successfully deleted
I20240502 01:53:30.671790 15667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdf6095f84aa497ba5db95d55c010085. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:30.684569 14952 raft_consensus.cc:1261] T daed3123a96a496da83a6110e6943264 P 7e74a3d5df6a483c95c9dabfc99577bc [term 1 FOLLOWER]: Refusing update from remote peer b4abc88ed2e54830ae3a317629916117: Log matching property violated. Preceding OpId in replica: term: 1 index: 81. Preceding OpId from leader: term: 1 index: 82. (index mismatch)
I20240502 01:53:30.692080 15667 tablet_copy_client.cc:783] T bdf6095f84aa497ba5db95d55c010085 P ce90a4bd06ba41dd8e5aa227de9780a5: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:30.692463 15667 tablet_copy_client.cc:647] T bdf6095f84aa497ba5db95d55c010085 P ce90a4bd06ba41dd8e5aa227de9780a5: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:30.695547 15667 tablet_copy_client.cc:523] T bdf6095f84aa497ba5db95d55c010085 P ce90a4bd06ba41dd8e5aa227de9780a5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:30.695488 15504 consensus_queue.cc:1035] T daed3123a96a496da83a6110e6943264 P b4abc88ed2e54830ae3a317629916117 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 82, Last known committed idx: 81, Time since last communication: 0.000s
I20240502 01:53:30.709254 15683 ts_tablet_manager.cc:902] T d38a05f7ffa94cc5abc50a76a83692b5 P 9e4fa889fab047baa5bd3dcd77b2ca8f: Initiating tablet copy from peer ce90a4bd06ba41dd8e5aa227de9780a5 (127.9.176.196:41329)
I20240502 01:53:30.711683 15667 tablet_bootstrap.cc:492] T bdf6095f84aa497ba5db95d55c010085 P ce90a4bd06ba41dd8e5aa227de9780a5: Bootstrap starting.
I20240502 01:53:30.719367 15625 raft_consensus.cc:2932] T daed3123a96a496da83a6110e6943264 P b4abc88ed2e54830ae3a317629916117 [term 1 LEADER]: Committing config change with OpId 1.82: config changed from index 81 to 82, VOTER 8446b7a7ea67486288d174eea7a3b3c6 (127.9.176.194) evicted. New config: { opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } } peers { permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: false } } }
I20240502 01:53:30.725117 13312 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet daed3123a96a496da83a6110e6943264 with cas_config_opid_index 81: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:53:30.727218 13325 catalog_manager.cc:5484] T daed3123a96a496da83a6110e6943264 P b4abc88ed2e54830ae3a317629916117 reported cstate change: config changed from index 81 to 82, VOTER 8446b7a7ea67486288d174eea7a3b3c6 (127.9.176.194) evicted. New cstate: current_term: 1 leader_uuid: "b4abc88ed2e54830ae3a317629916117" committed_config { opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:53:30.738909 14954 raft_consensus.cc:2932] T daed3123a96a496da83a6110e6943264 P 7e74a3d5df6a483c95c9dabfc99577bc [term 1 FOLLOWER]: Committing config change with OpId 1.82: config changed from index 81 to 82, VOTER 8446b7a7ea67486288d174eea7a3b3c6 (127.9.176.194) evicted. New config: { opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } } peers { permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: false } } }
I20240502 01:53:30.756726 13584 tablet_service.cc:1514] Processing DeleteTablet for tablet daed3123a96a496da83a6110e6943264 with delete_type TABLET_DATA_TOMBSTONED (TS 8446b7a7ea67486288d174eea7a3b3c6 not found in new config with opid_index 82) from {username='slave'} at 127.0.0.1:41580
I20240502 01:53:30.757197 15674 tablet_replica.cc:330] T daed3123a96a496da83a6110e6943264 P 8446b7a7ea67486288d174eea7a3b3c6: stopping tablet replica
I20240502 01:53:30.757443 15674 raft_consensus.cc:2229] T daed3123a96a496da83a6110e6943264 P 8446b7a7ea67486288d174eea7a3b3c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:30.757627 15638 ts_tablet_manager.cc:1888] T edf0aa94bfb94d499015897713ed94fa P 5359629f948641f4b56e35ad6b6a15c4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.71
I20240502 01:53:30.757781 15638 log.cc:1196] T edf0aa94bfb94d499015897713ed94fa P 5359629f948641f4b56e35ad6b6a15c4: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/wal/wals/edf0aa94bfb94d499015897713ed94fa
I20240502 01:53:30.758062 15674 raft_consensus.cc:2258] T daed3123a96a496da83a6110e6943264 P 8446b7a7ea67486288d174eea7a3b3c6 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240502 01:53:30.765134 13402 consensus_peers.cc:485] T b38a1fb4d1694c609a932e8816fc3514 P 5359629f948641f4b56e35ad6b6a15c4 -> Peer 7e74a3d5df6a483c95c9dabfc99577bc (127.9.176.197:35757): Couldn't send request to peer 7e74a3d5df6a483c95c9dabfc99577bc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b38a1fb4d1694c609a932e8816fc3514. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:30.766136 13313 catalog_manager.cc:4830] TS 5359629f948641f4b56e35ad6b6a15c4 (127.9.176.193:33849): tablet edf0aa94bfb94d499015897713ed94fa (table rebalance_test_table_1 [id=8ebdfe08a0b244819813a774f9d721fc]) successfully deleted
I20240502 01:53:30.768765 15674 ts_tablet_manager.cc:1875] T daed3123a96a496da83a6110e6943264 P 8446b7a7ea67486288d174eea7a3b3c6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:53:30.778287 15674 ts_tablet_manager.cc:1888] T daed3123a96a496da83a6110e6943264 P 8446b7a7ea67486288d174eea7a3b3c6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.81
I20240502 01:53:30.778594 15674 log.cc:1196] T daed3123a96a496da83a6110e6943264 P 8446b7a7ea67486288d174eea7a3b3c6: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/wal/wals/daed3123a96a496da83a6110e6943264
I20240502 01:53:30.782111 13310 catalog_manager.cc:4830] TS 8446b7a7ea67486288d174eea7a3b3c6 (127.9.176.194:34851): tablet daed3123a96a496da83a6110e6943264 (table rebalance_test_table_0 [id=42b769e9abfb4c36b2891f39a41750d2]) successfully deleted
I20240502 01:53:30.817065 15683 tablet_copy_client.cc:308] T d38a05f7ffa94cc5abc50a76a83692b5 P 9e4fa889fab047baa5bd3dcd77b2ca8f: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.196:41329
I20240502 01:53:30.928608 15652 tablet_bootstrap.cc:492] T f2af8f5d5e784e1988a957666844db6b P 7505fefeaf804ed8aeb3672f386ab959: Bootstrap replayed 1/1 log segments. Stats: ops{read=66 overwritten=0 applied=65 ignored=0} inserts{seen=696 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 01:53:30.929212 15652 tablet_bootstrap.cc:492] T f2af8f5d5e784e1988a957666844db6b P 7505fefeaf804ed8aeb3672f386ab959: Bootstrap complete.
I20240502 01:53:30.929527 15652 ts_tablet_manager.cc:1367] T f2af8f5d5e784e1988a957666844db6b P 7505fefeaf804ed8aeb3672f386ab959: Time spent bootstrapping tablet: real 0.423s	user 0.141s	sys 0.012s
I20240502 01:53:30.931138 15652 raft_consensus.cc:348] T f2af8f5d5e784e1988a957666844db6b P 7505fefeaf804ed8aeb3672f386ab959 [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 61 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } attrs { replace: true } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } } peers { permanent_uuid: "7505fefeaf804ed8aeb3672f386ab959" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 38645 } attrs { promote: true } }
I20240502 01:53:30.938550 15652 raft_consensus.cc:729] T f2af8f5d5e784e1988a957666844db6b P 7505fefeaf804ed8aeb3672f386ab959 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7505fefeaf804ed8aeb3672f386ab959, State: Initialized, Role: LEARNER
I20240502 01:53:30.938835 15652 consensus_queue.cc:260] T f2af8f5d5e784e1988a957666844db6b P 7505fefeaf804ed8aeb3672f386ab959 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 65, Last appended: 1.66, Last appended by leader: 66, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 61 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } attrs { replace: true } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } } peers { permanent_uuid: "7505fefeaf804ed8aeb3672f386ab959" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 38645 } attrs { promote: true } }
I20240502 01:53:30.946259 15652 ts_tablet_manager.cc:1398] T f2af8f5d5e784e1988a957666844db6b P 7505fefeaf804ed8aeb3672f386ab959: Time spent starting tablet: real 0.017s	user 0.008s	sys 0.007s
I20240502 01:53:30.947530 13756 tablet_copy_service.cc:342] P b4abc88ed2e54830ae3a317629916117: Request end of tablet copy session 7505fefeaf804ed8aeb3672f386ab959-f2af8f5d5e784e1988a957666844db6b received from {username='slave'} at 127.9.176.199:55115
I20240502 01:53:30.947757 13756 tablet_copy_service.cc:433] P b4abc88ed2e54830ae3a317629916117: ending tablet copy session 7505fefeaf804ed8aeb3672f386ab959-f2af8f5d5e784e1988a957666844db6b on tablet f2af8f5d5e784e1988a957666844db6b with peer 7505fefeaf804ed8aeb3672f386ab959
W20240502 01:53:30.998112 13402 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.176.194:34851, user_credentials={real_user=slave}} blocked reactor thread for 77018.3us
I20240502 01:53:31.095324 13471 consensus_queue.cc:237] T 01fb93c5dab3401fa5a103ead4dd9708 P 5359629f948641f4b56e35ad6b6a15c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 85, Committed index: 85, Last appended: 1.85, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 86 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } attrs { replace: true } } peers { permanent_uuid: "7505fefeaf804ed8aeb3672f386ab959" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 38645 } attrs { promote: true } }
I20240502 01:53:31.104635 13599 raft_consensus.cc:1261] T 01fb93c5dab3401fa5a103ead4dd9708 P 8446b7a7ea67486288d174eea7a3b3c6 [term 1 FOLLOWER]: Refusing update from remote peer 5359629f948641f4b56e35ad6b6a15c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 85. Preceding OpId from leader: term: 1 index: 86. (index mismatch)
I20240502 01:53:31.106403 14841 tablet_copy_service.cc:140] P ce90a4bd06ba41dd8e5aa227de9780a5: Received BeginTabletCopySession request for tablet d38a05f7ffa94cc5abc50a76a83692b5 from peer 9e4fa889fab047baa5bd3dcd77b2ca8f ({username='slave'} at 127.9.176.198:36377)
I20240502 01:53:31.106738 14841 tablet_copy_service.cc:161] P ce90a4bd06ba41dd8e5aa227de9780a5: Beginning new tablet copy session on tablet d38a05f7ffa94cc5abc50a76a83692b5 from peer 9e4fa889fab047baa5bd3dcd77b2ca8f at {username='slave'} at 127.9.176.198:36377: session id = 9e4fa889fab047baa5bd3dcd77b2ca8f-d38a05f7ffa94cc5abc50a76a83692b5
W20240502 01:53:31.111109 13405 consensus_peers.cc:485] T 01fb93c5dab3401fa5a103ead4dd9708 P 5359629f948641f4b56e35ad6b6a15c4 -> Peer 7505fefeaf804ed8aeb3672f386ab959 (127.9.176.199:38645): Couldn't send request to peer 7505fefeaf804ed8aeb3672f386ab959. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 01fb93c5dab3401fa5a103ead4dd9708. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:31.111701 14841 tablet_copy_source_session.cc:205] T d38a05f7ffa94cc5abc50a76a83692b5 P ce90a4bd06ba41dd8e5aa227de9780a5: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:31.114004 15356 consensus_queue.cc:1035] T 01fb93c5dab3401fa5a103ead4dd9708 P 5359629f948641f4b56e35ad6b6a15c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 86, Last known committed idx: 85, Time since last communication: 0.000s
I20240502 01:53:31.114518 15683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d38a05f7ffa94cc5abc50a76a83692b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:31.124192 15360 raft_consensus.cc:2932] T 01fb93c5dab3401fa5a103ead4dd9708 P 5359629f948641f4b56e35ad6b6a15c4 [term 1 LEADER]: Committing config change with OpId 1.86: config changed from index -1 to 86, NON_VOTER 7505fefeaf804ed8aeb3672f386ab959 (127.9.176.199) added. New config: { opid_index: 86 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } attrs { replace: true } } peers { permanent_uuid: "7505fefeaf804ed8aeb3672f386ab959" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 38645 } attrs { promote: true } } }
I20240502 01:53:31.133179 13325 catalog_manager.cc:5484] T 01fb93c5dab3401fa5a103ead4dd9708 P 5359629f948641f4b56e35ad6b6a15c4 reported cstate change: config changed from index -1 to 86, NON_VOTER 7505fefeaf804ed8aeb3672f386ab959 (127.9.176.199) added. New cstate: current_term: 1 leader_uuid: "5359629f948641f4b56e35ad6b6a15c4" committed_config { opid_index: 86 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7505fefeaf804ed8aeb3672f386ab959" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 38645 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:31.135166 15683 tablet_copy_client.cc:783] T d38a05f7ffa94cc5abc50a76a83692b5 P 9e4fa889fab047baa5bd3dcd77b2ca8f: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:31.135680 15683 tablet_copy_client.cc:647] T d38a05f7ffa94cc5abc50a76a83692b5 P 9e4fa889fab047baa5bd3dcd77b2ca8f: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:31.142935 15683 tablet_copy_client.cc:523] T d38a05f7ffa94cc5abc50a76a83692b5 P 9e4fa889fab047baa5bd3dcd77b2ca8f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:31.146935 15683 tablet_bootstrap.cc:492] T d38a05f7ffa94cc5abc50a76a83692b5 P 9e4fa889fab047baa5bd3dcd77b2ca8f: Bootstrap starting.
I20240502 01:53:31.167249 15683 tablet_bootstrap.cc:492] T d38a05f7ffa94cc5abc50a76a83692b5 P 9e4fa889fab047baa5bd3dcd77b2ca8f: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:53:31.170208 15683 tablet_bootstrap.cc:492] T d38a05f7ffa94cc5abc50a76a83692b5 P 9e4fa889fab047baa5bd3dcd77b2ca8f: Bootstrap complete.
I20240502 01:53:31.172842 15683 ts_tablet_manager.cc:1367] T d38a05f7ffa94cc5abc50a76a83692b5 P 9e4fa889fab047baa5bd3dcd77b2ca8f: Time spent bootstrapping tablet: real 0.026s	user 0.013s	sys 0.002s
I20240502 01:53:31.174152 15683 raft_consensus.cc:348] T d38a05f7ffa94cc5abc50a76a83692b5 P 9e4fa889fab047baa5bd3dcd77b2ca8f [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } attrs { replace: true } } peers { permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 41329 } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: true } }
I20240502 01:53:31.174571 15683 raft_consensus.cc:729] T d38a05f7ffa94cc5abc50a76a83692b5 P 9e4fa889fab047baa5bd3dcd77b2ca8f [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 9e4fa889fab047baa5bd3dcd77b2ca8f, State: Initialized, Role: LEARNER
I20240502 01:53:31.174919 15683 consensus_queue.cc:260] T d38a05f7ffa94cc5abc50a76a83692b5 P 9e4fa889fab047baa5bd3dcd77b2ca8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } attrs { replace: true } } peers { permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 41329 } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: true } }
I20240502 01:53:31.176003 13599 raft_consensus.cc:2932] T 01fb93c5dab3401fa5a103ead4dd9708 P 8446b7a7ea67486288d174eea7a3b3c6 [term 1 FOLLOWER]: Committing config change with OpId 1.86: config changed from index -1 to 86, NON_VOTER 7505fefeaf804ed8aeb3672f386ab959 (127.9.176.199) added. New config: { opid_index: 86 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } attrs { replace: true } } peers { permanent_uuid: "7505fefeaf804ed8aeb3672f386ab959" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 38645 } attrs { promote: true } } }
I20240502 01:53:31.182727 15683 ts_tablet_manager.cc:1398] T d38a05f7ffa94cc5abc50a76a83692b5 P 9e4fa889fab047baa5bd3dcd77b2ca8f: Time spent starting tablet: real 0.010s	user 0.003s	sys 0.005s
I20240502 01:53:31.184774 14841 tablet_copy_service.cc:342] P ce90a4bd06ba41dd8e5aa227de9780a5: Request end of tablet copy session 9e4fa889fab047baa5bd3dcd77b2ca8f-d38a05f7ffa94cc5abc50a76a83692b5 received from {username='slave'} at 127.9.176.198:36377
I20240502 01:53:31.185007 14841 tablet_copy_service.cc:433] P ce90a4bd06ba41dd8e5aa227de9780a5: ending tablet copy session 9e4fa889fab047baa5bd3dcd77b2ca8f-d38a05f7ffa94cc5abc50a76a83692b5 on tablet d38a05f7ffa94cc5abc50a76a83692b5 with peer 9e4fa889fab047baa5bd3dcd77b2ca8f
I20240502 01:53:31.203071 15227 raft_consensus.cc:1203] T f2af8f5d5e784e1988a957666844db6b P 7505fefeaf804ed8aeb3672f386ab959 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.60->[1.61-1.73]   Dedup: 1.66->[1.67-1.73]
I20240502 01:53:31.247829 15701 ts_tablet_manager.cc:902] T b38a1fb4d1694c609a932e8816fc3514 P 7e74a3d5df6a483c95c9dabfc99577bc: Initiating tablet copy from peer 5359629f948641f4b56e35ad6b6a15c4 (127.9.176.193:33849)
I20240502 01:53:31.257293 15087 raft_consensus.cc:1203] T d38a05f7ffa94cc5abc50a76a83692b5 P 9e4fa889fab047baa5bd3dcd77b2ca8f [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3]   Dedup: 2.3->[]
I20240502 01:53:31.269312 15667 tablet_bootstrap.cc:492] T bdf6095f84aa497ba5db95d55c010085 P ce90a4bd06ba41dd8e5aa227de9780a5: Bootstrap replayed 1/1 log segments. Stats: ops{read=79 overwritten=0 applied=79 ignored=0} inserts{seen=751 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:53:31.269933 15667 tablet_bootstrap.cc:492] T bdf6095f84aa497ba5db95d55c010085 P ce90a4bd06ba41dd8e5aa227de9780a5: Bootstrap complete.
I20240502 01:53:31.270321 15667 ts_tablet_manager.cc:1367] T bdf6095f84aa497ba5db95d55c010085 P ce90a4bd06ba41dd8e5aa227de9780a5: Time spent bootstrapping tablet: real 0.571s	user 0.218s	sys 0.085s
I20240502 01:53:31.271337 15667 raft_consensus.cc:348] T bdf6095f84aa497ba5db95d55c010085 P ce90a4bd06ba41dd8e5aa227de9780a5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } attrs { replace: true } } peers { permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 41329 } attrs { promote: true } }
I20240502 01:53:31.271663 15667 raft_consensus.cc:729] T bdf6095f84aa497ba5db95d55c010085 P ce90a4bd06ba41dd8e5aa227de9780a5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ce90a4bd06ba41dd8e5aa227de9780a5, State: Initialized, Role: LEARNER
I20240502 01:53:31.271920 15667 consensus_queue.cc:260] T bdf6095f84aa497ba5db95d55c010085 P ce90a4bd06ba41dd8e5aa227de9780a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 79, Last appended: 1.79, Last appended by leader: 79, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } attrs { replace: true } } peers { permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 41329 } attrs { promote: true } }
I20240502 01:53:31.273633 15667 ts_tablet_manager.cc:1398] T bdf6095f84aa497ba5db95d55c010085 P ce90a4bd06ba41dd8e5aa227de9780a5: Time spent starting tablet: real 0.003s	user 0.003s	sys 0.001s
I20240502 01:53:31.275560 13491 tablet_copy_service.cc:342] P 5359629f948641f4b56e35ad6b6a15c4: Request end of tablet copy session ce90a4bd06ba41dd8e5aa227de9780a5-bdf6095f84aa497ba5db95d55c010085 received from {username='slave'} at 127.9.176.196:54151
I20240502 01:53:31.275776 13491 tablet_copy_service.cc:433] P 5359629f948641f4b56e35ad6b6a15c4: ending tablet copy session ce90a4bd06ba41dd8e5aa227de9780a5-bdf6095f84aa497ba5db95d55c010085 on tablet bdf6095f84aa497ba5db95d55c010085 with peer ce90a4bd06ba41dd8e5aa227de9780a5
I20240502 01:53:31.302520 15701 tablet_copy_client.cc:308] T b38a1fb4d1694c609a932e8816fc3514 P 7e74a3d5df6a483c95c9dabfc99577bc: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.193:33849
I20240502 01:53:31.307487 13598 consensus_queue.cc:237] T d4511dd1019c4bdcbf9fc7e4bfe2afb7 P 8446b7a7ea67486288d174eea7a3b3c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 87, Committed index: 87, Last appended: 1.87, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 88 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: true } }
I20240502 01:53:31.319298 13734 raft_consensus.cc:1261] T d4511dd1019c4bdcbf9fc7e4bfe2afb7 P b4abc88ed2e54830ae3a317629916117 [term 1 FOLLOWER]: Refusing update from remote peer 8446b7a7ea67486288d174eea7a3b3c6: Log matching property violated. Preceding OpId in replica: term: 1 index: 87. Preceding OpId from leader: term: 1 index: 88. (index mismatch)
I20240502 01:53:31.320053 15498 consensus_queue.cc:1035] T d4511dd1019c4bdcbf9fc7e4bfe2afb7 P 8446b7a7ea67486288d174eea7a3b3c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 88, Last known committed idx: 87, Time since last communication: 0.000s
I20240502 01:53:31.322803 15498 raft_consensus.cc:2932] T d4511dd1019c4bdcbf9fc7e4bfe2afb7 P 8446b7a7ea67486288d174eea7a3b3c6 [term 1 LEADER]: Committing config change with OpId 1.88: config changed from index -1 to 88, NON_VOTER 9e4fa889fab047baa5bd3dcd77b2ca8f (127.9.176.198) added. New config: { opid_index: 88 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: true } } }
I20240502 01:53:31.329309 13734 raft_consensus.cc:2932] T d4511dd1019c4bdcbf9fc7e4bfe2afb7 P b4abc88ed2e54830ae3a317629916117 [term 1 FOLLOWER]: Committing config change with OpId 1.88: config changed from index -1 to 88, NON_VOTER 9e4fa889fab047baa5bd3dcd77b2ca8f (127.9.176.198) added. New config: { opid_index: 88 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: true } } }
I20240502 01:53:31.341017 13326 catalog_manager.cc:5484] T d4511dd1019c4bdcbf9fc7e4bfe2afb7 P b4abc88ed2e54830ae3a317629916117 reported cstate change: config changed from index -1 to 88, NON_VOTER 9e4fa889fab047baa5bd3dcd77b2ca8f (127.9.176.198) added. New cstate: current_term: 1 leader_uuid: "8446b7a7ea67486288d174eea7a3b3c6" committed_config { opid_index: 88 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: true } } }
I20240502 01:53:31.353649 15625 raft_consensus.cc:1050] T f2af8f5d5e784e1988a957666844db6b P b4abc88ed2e54830ae3a317629916117: attempting to promote NON_VOTER 7505fefeaf804ed8aeb3672f386ab959 to VOTER
I20240502 01:53:31.354579 15625 consensus_queue.cc:237] T f2af8f5d5e784e1988a957666844db6b P b4abc88ed2e54830ae3a317629916117 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 73, Committed index: 73, Last appended: 1.74, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 75 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } attrs { replace: true } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } } peers { permanent_uuid: "7505fefeaf804ed8aeb3672f386ab959" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 38645 } attrs { promote: false } }
I20240502 01:53:31.363229 15227 raft_consensus.cc:1261] T f2af8f5d5e784e1988a957666844db6b P 7505fefeaf804ed8aeb3672f386ab959 [term 1 LEARNER]: Refusing update from remote peer b4abc88ed2e54830ae3a317629916117: Log matching property violated. Preceding OpId in replica: term: 1 index: 74. Preceding OpId from leader: term: 1 index: 75. (index mismatch)
I20240502 01:53:31.364176 15625 consensus_queue.cc:1035] T f2af8f5d5e784e1988a957666844db6b P b4abc88ed2e54830ae3a317629916117 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7505fefeaf804ed8aeb3672f386ab959" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 38645 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 75, Last known committed idx: 73, Time since last communication: 0.000s
I20240502 01:53:31.373795 13598 raft_consensus.cc:1261] T f2af8f5d5e784e1988a957666844db6b P 8446b7a7ea67486288d174eea7a3b3c6 [term 1 FOLLOWER]: Refusing update from remote peer b4abc88ed2e54830ae3a317629916117: Log matching property violated. Preceding OpId in replica: term: 1 index: 74. Preceding OpId from leader: term: 1 index: 75. (index mismatch)
I20240502 01:53:31.374755 15625 consensus_queue.cc:1035] T f2af8f5d5e784e1988a957666844db6b P b4abc88ed2e54830ae3a317629916117 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 75, Last known committed idx: 73, Time since last communication: 0.000s
I20240502 01:53:31.377470 15625 raft_consensus.cc:2932] T f2af8f5d5e784e1988a957666844db6b P b4abc88ed2e54830ae3a317629916117 [term 1 LEADER]: Committing config change with OpId 1.75: config changed from index 61 to 75, 7505fefeaf804ed8aeb3672f386ab959 (127.9.176.199) changed from NON_VOTER to VOTER. New config: { opid_index: 75 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } attrs { replace: true } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } } peers { permanent_uuid: "7505fefeaf804ed8aeb3672f386ab959" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 38645 } attrs { promote: false } } }
I20240502 01:53:31.381558 15228 raft_consensus.cc:2932] T f2af8f5d5e784e1988a957666844db6b P 7505fefeaf804ed8aeb3672f386ab959 [term 1 FOLLOWER]: Committing config change with OpId 1.75: config changed from index 61 to 75, 7505fefeaf804ed8aeb3672f386ab959 (127.9.176.199) changed from NON_VOTER to VOTER. New config: { opid_index: 75 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } attrs { replace: true } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } } peers { permanent_uuid: "7505fefeaf804ed8aeb3672f386ab959" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 38645 } attrs { promote: false } } }
I20240502 01:53:31.383966 13598 raft_consensus.cc:2932] T f2af8f5d5e784e1988a957666844db6b P 8446b7a7ea67486288d174eea7a3b3c6 [term 1 FOLLOWER]: Committing config change with OpId 1.75: config changed from index 61 to 75, 7505fefeaf804ed8aeb3672f386ab959 (127.9.176.199) changed from NON_VOTER to VOTER. New config: { opid_index: 75 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } attrs { replace: true } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } } peers { permanent_uuid: "7505fefeaf804ed8aeb3672f386ab959" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 38645 } attrs { promote: false } } }
I20240502 01:53:31.396726 13325 catalog_manager.cc:5484] T f2af8f5d5e784e1988a957666844db6b P 8446b7a7ea67486288d174eea7a3b3c6 reported cstate change: config changed from index 61 to 75, 7505fefeaf804ed8aeb3672f386ab959 (127.9.176.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b4abc88ed2e54830ae3a317629916117" committed_config { opid_index: 75 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } attrs { replace: true } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } } peers { permanent_uuid: "7505fefeaf804ed8aeb3672f386ab959" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 38645 } attrs { promote: false } } }
I20240502 01:53:31.458325 14820 raft_consensus.cc:1203] T bdf6095f84aa497ba5db95d55c010085 P ce90a4bd06ba41dd8e5aa227de9780a5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.71->[1.72-1.87]   Dedup: 1.79->[1.80-1.87]
I20240502 01:53:31.481209 13599 consensus_queue.cc:237] T 930645182ab748d7b864fece1e858b3f P 8446b7a7ea67486288d174eea7a3b3c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 70, Committed index: 70, Last appended: 1.70, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: true } }
I20240502 01:53:31.486114 13734 raft_consensus.cc:1261] T 930645182ab748d7b864fece1e858b3f P b4abc88ed2e54830ae3a317629916117 [term 1 FOLLOWER]: Refusing update from remote peer 8446b7a7ea67486288d174eea7a3b3c6: Log matching property violated. Preceding OpId in replica: term: 1 index: 70. Preceding OpId from leader: term: 1 index: 71. (index mismatch)
I20240502 01:53:31.490077 15498 consensus_queue.cc:1035] T 930645182ab748d7b864fece1e858b3f P 8446b7a7ea67486288d174eea7a3b3c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 71, Last known committed idx: 70, Time since last communication: 0.000s
I20240502 01:53:31.493396 15449 raft_consensus.cc:2932] T 930645182ab748d7b864fece1e858b3f P 8446b7a7ea67486288d174eea7a3b3c6 [term 1 LEADER]: Committing config change with OpId 1.71: config changed from index -1 to 71, NON_VOTER 9e4fa889fab047baa5bd3dcd77b2ca8f (127.9.176.198) added. New config: { opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: true } } }
I20240502 01:53:31.503159 13734 raft_consensus.cc:2932] T 930645182ab748d7b864fece1e858b3f P b4abc88ed2e54830ae3a317629916117 [term 1 FOLLOWER]: Committing config change with OpId 1.71: config changed from index -1 to 71, NON_VOTER 9e4fa889fab047baa5bd3dcd77b2ca8f (127.9.176.198) added. New config: { opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: true } } }
I20240502 01:53:31.508180 13735 consensus_queue.cc:237] T f2af8f5d5e784e1988a957666844db6b P b4abc88ed2e54830ae3a317629916117 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 77, Committed index: 77, Last appended: 1.77, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 78 OBSOLETE_local: false peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } } peers { permanent_uuid: "7505fefeaf804ed8aeb3672f386ab959" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 38645 } attrs { promote: false } }
I20240502 01:53:31.518919 15226 raft_consensus.cc:1261] T f2af8f5d5e784e1988a957666844db6b P 7505fefeaf804ed8aeb3672f386ab959 [term 1 FOLLOWER]: Refusing update from remote peer b4abc88ed2e54830ae3a317629916117: Log matching property violated. Preceding OpId in replica: term: 1 index: 77. Preceding OpId from leader: term: 1 index: 78. (index mismatch)
I20240502 01:53:31.527204 15369 consensus_queue.cc:1035] T f2af8f5d5e784e1988a957666844db6b P b4abc88ed2e54830ae3a317629916117 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7505fefeaf804ed8aeb3672f386ab959" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 38645 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 78, Last known committed idx: 77, Time since last communication: 0.000s
I20240502 01:53:31.530405 13325 catalog_manager.cc:5484] T 930645182ab748d7b864fece1e858b3f P 8446b7a7ea67486288d174eea7a3b3c6 reported cstate change: config changed from index -1 to 71, NON_VOTER 9e4fa889fab047baa5bd3dcd77b2ca8f (127.9.176.198) added. New cstate: current_term: 1 leader_uuid: "8446b7a7ea67486288d174eea7a3b3c6" committed_config { opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:31.555023 15732 ts_tablet_manager.cc:902] T 01fb93c5dab3401fa5a103ead4dd9708 P 7505fefeaf804ed8aeb3672f386ab959: Initiating tablet copy from peer 5359629f948641f4b56e35ad6b6a15c4 (127.9.176.193:33849)
I20240502 01:53:31.560652 13491 tablet_copy_service.cc:140] P 5359629f948641f4b56e35ad6b6a15c4: Received BeginTabletCopySession request for tablet b38a1fb4d1694c609a932e8816fc3514 from peer 7e74a3d5df6a483c95c9dabfc99577bc ({username='slave'} at 127.9.176.197:57335)
I20240502 01:53:31.560847 13491 tablet_copy_service.cc:161] P 5359629f948641f4b56e35ad6b6a15c4: Beginning new tablet copy session on tablet b38a1fb4d1694c609a932e8816fc3514 from peer 7e74a3d5df6a483c95c9dabfc99577bc at {username='slave'} at 127.9.176.197:57335: session id = 7e74a3d5df6a483c95c9dabfc99577bc-b38a1fb4d1694c609a932e8816fc3514
I20240502 01:53:31.563936 13491 tablet_copy_source_session.cc:205] T b38a1fb4d1694c609a932e8816fc3514 P 5359629f948641f4b56e35ad6b6a15c4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:31.565817 15701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b38a1fb4d1694c609a932e8816fc3514. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:31.569546 15369 raft_consensus.cc:2932] T f2af8f5d5e784e1988a957666844db6b P b4abc88ed2e54830ae3a317629916117 [term 1 LEADER]: Committing config change with OpId 1.78: config changed from index 75 to 78, VOTER 8446b7a7ea67486288d174eea7a3b3c6 (127.9.176.194) evicted. New config: { opid_index: 78 OBSOLETE_local: false peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } } peers { permanent_uuid: "7505fefeaf804ed8aeb3672f386ab959" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 38645 } attrs { promote: false } } }
I20240502 01:53:31.573678 15701 tablet_copy_client.cc:783] T b38a1fb4d1694c609a932e8816fc3514 P 7e74a3d5df6a483c95c9dabfc99577bc: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:31.573990 15701 tablet_copy_client.cc:647] T b38a1fb4d1694c609a932e8816fc3514 P 7e74a3d5df6a483c95c9dabfc99577bc: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:31.575944 15228 raft_consensus.cc:2932] T f2af8f5d5e784e1988a957666844db6b P 7505fefeaf804ed8aeb3672f386ab959 [term 1 FOLLOWER]: Committing config change with OpId 1.78: config changed from index 75 to 78, VOTER 8446b7a7ea67486288d174eea7a3b3c6 (127.9.176.194) evicted. New config: { opid_index: 78 OBSOLETE_local: false peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } } peers { permanent_uuid: "7505fefeaf804ed8aeb3672f386ab959" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 38645 } attrs { promote: false } } }
I20240502 01:53:31.579888 13312 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f2af8f5d5e784e1988a957666844db6b with cas_config_opid_index 75: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:53:31.580413 15701 tablet_copy_client.cc:523] T b38a1fb4d1694c609a932e8816fc3514 P 7e74a3d5df6a483c95c9dabfc99577bc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:31.584852 15701 tablet_bootstrap.cc:492] T b38a1fb4d1694c609a932e8816fc3514 P 7e74a3d5df6a483c95c9dabfc99577bc: Bootstrap starting.
I20240502 01:53:31.585840 13326 catalog_manager.cc:5484] T f2af8f5d5e784e1988a957666844db6b P b4abc88ed2e54830ae3a317629916117 reported cstate change: config changed from index 75 to 78, VOTER 8446b7a7ea67486288d174eea7a3b3c6 (127.9.176.194) evicted. New cstate: current_term: 1 leader_uuid: "b4abc88ed2e54830ae3a317629916117" committed_config { opid_index: 78 OBSOLETE_local: false peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7505fefeaf804ed8aeb3672f386ab959" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 38645 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240502 01:53:31.590133 13539 consensus_peers.cc:485] T d4511dd1019c4bdcbf9fc7e4bfe2afb7 P 8446b7a7ea67486288d174eea7a3b3c6 -> Peer 9e4fa889fab047baa5bd3dcd77b2ca8f (127.9.176.198:38527): Couldn't send request to peer 9e4fa889fab047baa5bd3dcd77b2ca8f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d4511dd1019c4bdcbf9fc7e4bfe2afb7. This is attempt 1: this message will repeat every 5th retry.
W20240502 01:53:31.590417 13539 consensus_peers.cc:485] T 930645182ab748d7b864fece1e858b3f P 8446b7a7ea67486288d174eea7a3b3c6 -> Peer 9e4fa889fab047baa5bd3dcd77b2ca8f (127.9.176.198:38527): Couldn't send request to peer 9e4fa889fab047baa5bd3dcd77b2ca8f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 930645182ab748d7b864fece1e858b3f. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:31.600837 13584 tablet_service.cc:1514] Processing DeleteTablet for tablet f2af8f5d5e784e1988a957666844db6b with delete_type TABLET_DATA_TOMBSTONED (TS 8446b7a7ea67486288d174eea7a3b3c6 not found in new config with opid_index 78) from {username='slave'} at 127.0.0.1:41580
I20240502 01:53:31.614877 15736 tablet_replica.cc:330] T f2af8f5d5e784e1988a957666844db6b P 8446b7a7ea67486288d174eea7a3b3c6: stopping tablet replica
I20240502 01:53:31.615341 15736 raft_consensus.cc:2229] T f2af8f5d5e784e1988a957666844db6b P 8446b7a7ea67486288d174eea7a3b3c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:31.615986 15736 raft_consensus.cc:2258] T f2af8f5d5e784e1988a957666844db6b P 8446b7a7ea67486288d174eea7a3b3c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:31.618674 15736 ts_tablet_manager.cc:1875] T f2af8f5d5e784e1988a957666844db6b P 8446b7a7ea67486288d174eea7a3b3c6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:53:31.648907 15711 raft_consensus.cc:1050] T d38a05f7ffa94cc5abc50a76a83692b5 P ce90a4bd06ba41dd8e5aa227de9780a5: attempting to promote NON_VOTER 9e4fa889fab047baa5bd3dcd77b2ca8f to VOTER
I20240502 01:53:31.650050 15711 consensus_queue.cc:237] T d38a05f7ffa94cc5abc50a76a83692b5 P ce90a4bd06ba41dd8e5aa227de9780a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } attrs { replace: true } } peers { permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 41329 } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false } }
I20240502 01:53:31.650352 15736 ts_tablet_manager.cc:1888] T f2af8f5d5e784e1988a957666844db6b P 8446b7a7ea67486288d174eea7a3b3c6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.77
I20240502 01:53:31.650473 15736 log.cc:1196] T f2af8f5d5e784e1988a957666844db6b P 8446b7a7ea67486288d174eea7a3b3c6: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/wal/wals/f2af8f5d5e784e1988a957666844db6b
I20240502 01:53:31.665213 15360 raft_consensus.cc:1050] T bdf6095f84aa497ba5db95d55c010085 P 5359629f948641f4b56e35ad6b6a15c4: attempting to promote NON_VOTER ce90a4bd06ba41dd8e5aa227de9780a5 to VOTER
I20240502 01:53:31.666097 15086 raft_consensus.cc:1261] T d38a05f7ffa94cc5abc50a76a83692b5 P 9e4fa889fab047baa5bd3dcd77b2ca8f [term 2 LEARNER]: Refusing update from remote peer ce90a4bd06ba41dd8e5aa227de9780a5: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 01:53:31.665987 15360 consensus_queue.cc:237] T bdf6095f84aa497ba5db95d55c010085 P 5359629f948641f4b56e35ad6b6a15c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 89, Committed index: 89, Last appended: 1.89, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 90 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } attrs { replace: true } } peers { permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 41329 } attrs { promote: false } }
I20240502 01:53:31.666798 15711 consensus_queue.cc:1035] T d38a05f7ffa94cc5abc50a76a83692b5 P ce90a4bd06ba41dd8e5aa227de9780a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:53:31.669564 13471 raft_consensus.cc:1261] T d38a05f7ffa94cc5abc50a76a83692b5 P 5359629f948641f4b56e35ad6b6a15c4 [term 2 FOLLOWER]: Refusing update from remote peer ce90a4bd06ba41dd8e5aa227de9780a5: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 01:53:31.670630 14821 raft_consensus.cc:1261] T bdf6095f84aa497ba5db95d55c010085 P ce90a4bd06ba41dd8e5aa227de9780a5 [term 1 LEARNER]: Refusing update from remote peer 5359629f948641f4b56e35ad6b6a15c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 88. Preceding OpId from leader: term: 1 index: 90. (index mismatch)
I20240502 01:53:31.671352 15356 consensus_queue.cc:1035] T bdf6095f84aa497ba5db95d55c010085 P 5359629f948641f4b56e35ad6b6a15c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 41329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 90, Last known committed idx: 88, Time since last communication: 0.000s
I20240502 01:53:31.674456 13310 catalog_manager.cc:4830] TS 8446b7a7ea67486288d174eea7a3b3c6 (127.9.176.194:34851): tablet f2af8f5d5e784e1988a957666844db6b (table rebalance_test_table_4 [id=7c97435ee22d42aaa3452b6e2abdaf75]) successfully deleted
I20240502 01:53:31.678275 14821 raft_consensus.cc:1203] T bdf6095f84aa497ba5db95d55c010085 P ce90a4bd06ba41dd8e5aa227de9780a5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.88->[1.89-1.90]   Dedup: 1.89->[1.90-1.90]
I20240502 01:53:31.678735 13598 raft_consensus.cc:1261] T bdf6095f84aa497ba5db95d55c010085 P 8446b7a7ea67486288d174eea7a3b3c6 [term 1 FOLLOWER]: Refusing update from remote peer 5359629f948641f4b56e35ad6b6a15c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 89. Preceding OpId from leader: term: 1 index: 90. (index mismatch)
I20240502 01:53:31.680366 15643 raft_consensus.cc:2932] T bdf6095f84aa497ba5db95d55c010085 P 5359629f948641f4b56e35ad6b6a15c4 [term 1 LEADER]: Committing config change with OpId 1.90: config changed from index 72 to 90, ce90a4bd06ba41dd8e5aa227de9780a5 (127.9.176.196) changed from NON_VOTER to VOTER. New config: { opid_index: 90 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } attrs { replace: true } } peers { permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 41329 } attrs { promote: false } } }
I20240502 01:53:31.683096 15711 consensus_queue.cc:1035] T d38a05f7ffa94cc5abc50a76a83692b5 P ce90a4bd06ba41dd8e5aa227de9780a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:53:31.686100 14820 raft_consensus.cc:2932] T bdf6095f84aa497ba5db95d55c010085 P ce90a4bd06ba41dd8e5aa227de9780a5 [term 1 FOLLOWER]: Committing config change with OpId 1.90: config changed from index 72 to 90, ce90a4bd06ba41dd8e5aa227de9780a5 (127.9.176.196) changed from NON_VOTER to VOTER. New config: { opid_index: 90 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } attrs { replace: true } } peers { permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 41329 } attrs { promote: false } } }
I20240502 01:53:31.693830 13326 catalog_manager.cc:5484] T bdf6095f84aa497ba5db95d55c010085 P 5359629f948641f4b56e35ad6b6a15c4 reported cstate change: config changed from index 72 to 90, ce90a4bd06ba41dd8e5aa227de9780a5 (127.9.176.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5359629f948641f4b56e35ad6b6a15c4" committed_config { opid_index: 90 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 41329 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:53:31.699748 15707 raft_consensus.cc:2932] T d38a05f7ffa94cc5abc50a76a83692b5 P ce90a4bd06ba41dd8e5aa227de9780a5 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 9e4fa889fab047baa5bd3dcd77b2ca8f (127.9.176.198) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } attrs { replace: true } } peers { permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 41329 } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false } } }
I20240502 01:53:31.700894 15356 consensus_queue.cc:1035] T bdf6095f84aa497ba5db95d55c010085 P 5359629f948641f4b56e35ad6b6a15c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 90, Last known committed idx: 89, Time since last communication: 0.000s
I20240502 01:53:31.711943 13317 catalog_manager.cc:5484] T d38a05f7ffa94cc5abc50a76a83692b5 P ce90a4bd06ba41dd8e5aa227de9780a5 reported cstate change: config changed from index 3 to 4, 9e4fa889fab047baa5bd3dcd77b2ca8f (127.9.176.198) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 41329 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:31.716406 15087 raft_consensus.cc:2932] T d38a05f7ffa94cc5abc50a76a83692b5 P 9e4fa889fab047baa5bd3dcd77b2ca8f [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 9e4fa889fab047baa5bd3dcd77b2ca8f (127.9.176.198) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } attrs { replace: true } } peers { permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 41329 } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false } } }
I20240502 01:53:31.720798 13470 raft_consensus.cc:2932] T d38a05f7ffa94cc5abc50a76a83692b5 P 5359629f948641f4b56e35ad6b6a15c4 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 9e4fa889fab047baa5bd3dcd77b2ca8f (127.9.176.198) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } attrs { replace: true } } peers { permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 41329 } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false } } }
I20240502 01:53:31.750664 13599 raft_consensus.cc:2932] T bdf6095f84aa497ba5db95d55c010085 P 8446b7a7ea67486288d174eea7a3b3c6 [term 1 FOLLOWER]: Committing config change with OpId 1.90: config changed from index 72 to 90, ce90a4bd06ba41dd8e5aa227de9780a5 (127.9.176.196) changed from NON_VOTER to VOTER. New config: { opid_index: 90 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } attrs { replace: true } } peers { permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 41329 } attrs { promote: false } } }
I20240502 01:53:31.786880 13470 consensus_queue.cc:237] T bdf6095f84aa497ba5db95d55c010085 P 5359629f948641f4b56e35ad6b6a15c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 93, Committed index: 93, Last appended: 1.93, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 41329 } attrs { promote: false } }
I20240502 01:53:31.790339 14820 raft_consensus.cc:1261] T bdf6095f84aa497ba5db95d55c010085 P ce90a4bd06ba41dd8e5aa227de9780a5 [term 1 FOLLOWER]: Refusing update from remote peer 5359629f948641f4b56e35ad6b6a15c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 93. Preceding OpId from leader: term: 1 index: 94. (index mismatch)
I20240502 01:53:31.792466 15360 consensus_queue.cc:1035] T bdf6095f84aa497ba5db95d55c010085 P 5359629f948641f4b56e35ad6b6a15c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 41329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 94, Last known committed idx: 93, Time since last communication: 0.000s
I20240502 01:53:31.795673 15360 raft_consensus.cc:2932] T bdf6095f84aa497ba5db95d55c010085 P 5359629f948641f4b56e35ad6b6a15c4 [term 1 LEADER]: Committing config change with OpId 1.94: config changed from index 90 to 94, VOTER 8446b7a7ea67486288d174eea7a3b3c6 (127.9.176.194) evicted. New config: { opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 41329 } attrs { promote: false } } }
I20240502 01:53:31.800575 13313 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet bdf6095f84aa497ba5db95d55c010085 with cas_config_opid_index 90: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:53:31.801190 14820 raft_consensus.cc:2932] T bdf6095f84aa497ba5db95d55c010085 P ce90a4bd06ba41dd8e5aa227de9780a5 [term 1 FOLLOWER]: Committing config change with OpId 1.94: config changed from index 90 to 94, VOTER 8446b7a7ea67486288d174eea7a3b3c6 (127.9.176.194) evicted. New config: { opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 41329 } attrs { promote: false } } }
I20240502 01:53:31.802242 13325 catalog_manager.cc:5484] T bdf6095f84aa497ba5db95d55c010085 P 5359629f948641f4b56e35ad6b6a15c4 reported cstate change: config changed from index 90 to 94, VOTER 8446b7a7ea67486288d174eea7a3b3c6 (127.9.176.194) evicted. New cstate: current_term: 1 leader_uuid: "5359629f948641f4b56e35ad6b6a15c4" committed_config { opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 41329 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:53:31.817000 15751 ts_tablet_manager.cc:902] T d4511dd1019c4bdcbf9fc7e4bfe2afb7 P 9e4fa889fab047baa5bd3dcd77b2ca8f: Initiating tablet copy from peer 8446b7a7ea67486288d174eea7a3b3c6 (127.9.176.194:34851)
I20240502 01:53:31.818815 15732 tablet_copy_client.cc:308] T 01fb93c5dab3401fa5a103ead4dd9708 P 7505fefeaf804ed8aeb3672f386ab959: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.193:33849
I20240502 01:53:31.831429 13584 tablet_service.cc:1514] Processing DeleteTablet for tablet bdf6095f84aa497ba5db95d55c010085 with delete_type TABLET_DATA_TOMBSTONED (TS 8446b7a7ea67486288d174eea7a3b3c6 not found in new config with opid_index 94) from {username='slave'} at 127.0.0.1:41580
I20240502 01:53:31.831913 15736 tablet_replica.cc:330] T bdf6095f84aa497ba5db95d55c010085 P 8446b7a7ea67486288d174eea7a3b3c6: stopping tablet replica
I20240502 01:53:31.832165 15736 raft_consensus.cc:2229] T bdf6095f84aa497ba5db95d55c010085 P 8446b7a7ea67486288d174eea7a3b3c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:31.844337 15736 raft_consensus.cc:2258] T bdf6095f84aa497ba5db95d55c010085 P 8446b7a7ea67486288d174eea7a3b3c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:31.845685 15736 ts_tablet_manager.cc:1875] T bdf6095f84aa497ba5db95d55c010085 P 8446b7a7ea67486288d174eea7a3b3c6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:53:31.852437 15736 ts_tablet_manager.cc:1888] T bdf6095f84aa497ba5db95d55c010085 P 8446b7a7ea67486288d174eea7a3b3c6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.90
I20240502 01:53:31.852720 15736 log.cc:1196] T bdf6095f84aa497ba5db95d55c010085 P 8446b7a7ea67486288d174eea7a3b3c6: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/wal/wals/bdf6095f84aa497ba5db95d55c010085
I20240502 01:53:31.860038 15751 tablet_copy_client.cc:308] T d4511dd1019c4bdcbf9fc7e4bfe2afb7 P 9e4fa889fab047baa5bd3dcd77b2ca8f: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.194:34851
I20240502 01:53:31.861820 14820 consensus_queue.cc:237] T d38a05f7ffa94cc5abc50a76a83692b5 P ce90a4bd06ba41dd8e5aa227de9780a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 41329 } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false } }
I20240502 01:53:31.878158 15701 tablet_bootstrap.cc:492] T b38a1fb4d1694c609a932e8816fc3514 P 7e74a3d5df6a483c95c9dabfc99577bc: Bootstrap replayed 1/1 log segments. Stats: ops{read=72 overwritten=0 applied=72 ignored=0} inserts{seen=703 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:53:31.878831 15701 tablet_bootstrap.cc:492] T b38a1fb4d1694c609a932e8816fc3514 P 7e74a3d5df6a483c95c9dabfc99577bc: Bootstrap complete.
I20240502 01:53:31.879267 15701 ts_tablet_manager.cc:1367] T b38a1fb4d1694c609a932e8816fc3514 P 7e74a3d5df6a483c95c9dabfc99577bc: Time spent bootstrapping tablet: real 0.295s	user 0.173s	sys 0.015s
I20240502 01:53:31.880420 15701 raft_consensus.cc:348] T b38a1fb4d1694c609a932e8816fc3514 P 7e74a3d5df6a483c95c9dabfc99577bc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 64 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: true } }
I20240502 01:53:31.880899 15701 raft_consensus.cc:729] T b38a1fb4d1694c609a932e8816fc3514 P 7e74a3d5df6a483c95c9dabfc99577bc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7e74a3d5df6a483c95c9dabfc99577bc, State: Initialized, Role: LEARNER
I20240502 01:53:31.881222 15701 consensus_queue.cc:260] T b38a1fb4d1694c609a932e8816fc3514 P 7e74a3d5df6a483c95c9dabfc99577bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 72, Last appended: 1.72, Last appended by leader: 72, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 64 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: true } }
I20240502 01:53:31.884299 15087 raft_consensus.cc:1261] T d38a05f7ffa94cc5abc50a76a83692b5 P 9e4fa889fab047baa5bd3dcd77b2ca8f [term 2 FOLLOWER]: Refusing update from remote peer ce90a4bd06ba41dd8e5aa227de9780a5: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 01:53:31.888814 15701 ts_tablet_manager.cc:1398] T b38a1fb4d1694c609a932e8816fc3514 P 7e74a3d5df6a483c95c9dabfc99577bc: Time spent starting tablet: real 0.009s	user 0.000s	sys 0.008s
I20240502 01:53:31.890337 13491 tablet_copy_service.cc:342] P 5359629f948641f4b56e35ad6b6a15c4: Request end of tablet copy session 7e74a3d5df6a483c95c9dabfc99577bc-b38a1fb4d1694c609a932e8816fc3514 received from {username='slave'} at 127.9.176.197:57335
I20240502 01:53:31.890532 13491 tablet_copy_service.cc:433] P 5359629f948641f4b56e35ad6b6a15c4: ending tablet copy session 7e74a3d5df6a483c95c9dabfc99577bc-b38a1fb4d1694c609a932e8816fc3514 on tablet b38a1fb4d1694c609a932e8816fc3514 with peer 7e74a3d5df6a483c95c9dabfc99577bc
I20240502 01:53:31.891378 15711 consensus_queue.cc:1035] T d38a05f7ffa94cc5abc50a76a83692b5 P ce90a4bd06ba41dd8e5aa227de9780a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:53:31.896512 13310 catalog_manager.cc:4830] TS 8446b7a7ea67486288d174eea7a3b3c6 (127.9.176.194:34851): tablet bdf6095f84aa497ba5db95d55c010085 (table rebalance_test_table_2 [id=c4a6875e444a446dbf9576f2a5d024c0]) successfully deleted
I20240502 01:53:31.906646 15760 ts_tablet_manager.cc:902] T 930645182ab748d7b864fece1e858b3f P 9e4fa889fab047baa5bd3dcd77b2ca8f: Initiating tablet copy from peer 8446b7a7ea67486288d174eea7a3b3c6 (127.9.176.194:34851)
I20240502 01:53:31.924250 15711 raft_consensus.cc:2932] T d38a05f7ffa94cc5abc50a76a83692b5 P ce90a4bd06ba41dd8e5aa227de9780a5 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 5359629f948641f4b56e35ad6b6a15c4 (127.9.176.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 41329 } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false } } }
I20240502 01:53:31.931912 13310 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d38a05f7ffa94cc5abc50a76a83692b5 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:53:31.936874 13318 catalog_manager.cc:5484] T d38a05f7ffa94cc5abc50a76a83692b5 P ce90a4bd06ba41dd8e5aa227de9780a5 reported cstate change: config changed from index 4 to 5, VOTER 5359629f948641f4b56e35ad6b6a15c4 (127.9.176.193) evicted. New cstate: current_term: 2 leader_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 41329 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:53:31.955479 13451 tablet_service.cc:1514] Processing DeleteTablet for tablet d38a05f7ffa94cc5abc50a76a83692b5 with delete_type TABLET_DATA_TOMBSTONED (TS 5359629f948641f4b56e35ad6b6a15c4 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:38182
I20240502 01:53:31.956969 13624 tablet_copy_service.cc:140] P 8446b7a7ea67486288d174eea7a3b3c6: Received BeginTabletCopySession request for tablet d4511dd1019c4bdcbf9fc7e4bfe2afb7 from peer 9e4fa889fab047baa5bd3dcd77b2ca8f ({username='slave'} at 127.9.176.198:41673)
I20240502 01:53:31.957209 13624 tablet_copy_service.cc:161] P 8446b7a7ea67486288d174eea7a3b3c6: Beginning new tablet copy session on tablet d4511dd1019c4bdcbf9fc7e4bfe2afb7 from peer 9e4fa889fab047baa5bd3dcd77b2ca8f at {username='slave'} at 127.9.176.198:41673: session id = 9e4fa889fab047baa5bd3dcd77b2ca8f-d4511dd1019c4bdcbf9fc7e4bfe2afb7
I20240502 01:53:31.963429 13624 tablet_copy_source_session.cc:205] T d4511dd1019c4bdcbf9fc7e4bfe2afb7 P 8446b7a7ea67486288d174eea7a3b3c6: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:31.970906 15087 raft_consensus.cc:2932] T d38a05f7ffa94cc5abc50a76a83692b5 P 9e4fa889fab047baa5bd3dcd77b2ca8f [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 5359629f948641f4b56e35ad6b6a15c4 (127.9.176.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 41329 } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false } } }
I20240502 01:53:31.989029 15760 tablet_copy_client.cc:308] T 930645182ab748d7b864fece1e858b3f P 9e4fa889fab047baa5bd3dcd77b2ca8f: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.194:34851
I20240502 01:53:31.991760 13491 tablet_copy_service.cc:140] P 5359629f948641f4b56e35ad6b6a15c4: Received BeginTabletCopySession request for tablet 01fb93c5dab3401fa5a103ead4dd9708 from peer 7505fefeaf804ed8aeb3672f386ab959 ({username='slave'} at 127.9.176.199:45881)
I20240502 01:53:31.992007 13491 tablet_copy_service.cc:161] P 5359629f948641f4b56e35ad6b6a15c4: Beginning new tablet copy session on tablet 01fb93c5dab3401fa5a103ead4dd9708 from peer 7505fefeaf804ed8aeb3672f386ab959 at {username='slave'} at 127.9.176.199:45881: session id = 7505fefeaf804ed8aeb3672f386ab959-01fb93c5dab3401fa5a103ead4dd9708
I20240502 01:53:31.995692 13491 tablet_copy_source_session.cc:205] T 01fb93c5dab3401fa5a103ead4dd9708 P 5359629f948641f4b56e35ad6b6a15c4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:31.997323 15732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01fb93c5dab3401fa5a103ead4dd9708. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:32.001611 15751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4511dd1019c4bdcbf9fc7e4bfe2afb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:32.004729 15732 tablet_copy_client.cc:783] T 01fb93c5dab3401fa5a103ead4dd9708 P 7505fefeaf804ed8aeb3672f386ab959: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:32.005070 15732 tablet_copy_client.cc:647] T 01fb93c5dab3401fa5a103ead4dd9708 P 7505fefeaf804ed8aeb3672f386ab959: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:32.007251 15732 tablet_copy_client.cc:523] T 01fb93c5dab3401fa5a103ead4dd9708 P 7505fefeaf804ed8aeb3672f386ab959: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:32.010887 15732 tablet_bootstrap.cc:492] T 01fb93c5dab3401fa5a103ead4dd9708 P 7505fefeaf804ed8aeb3672f386ab959: Bootstrap starting.
I20240502 01:53:32.014564 15751 tablet_copy_client.cc:783] T d4511dd1019c4bdcbf9fc7e4bfe2afb7 P 9e4fa889fab047baa5bd3dcd77b2ca8f: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:32.014916 15751 tablet_copy_client.cc:647] T d4511dd1019c4bdcbf9fc7e4bfe2afb7 P 9e4fa889fab047baa5bd3dcd77b2ca8f: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:32.025435 13624 tablet_copy_service.cc:140] P 8446b7a7ea67486288d174eea7a3b3c6: Received BeginTabletCopySession request for tablet 930645182ab748d7b864fece1e858b3f from peer 9e4fa889fab047baa5bd3dcd77b2ca8f ({username='slave'} at 127.9.176.198:41673)
I20240502 01:53:32.025643 13624 tablet_copy_service.cc:161] P 8446b7a7ea67486288d174eea7a3b3c6: Beginning new tablet copy session on tablet 930645182ab748d7b864fece1e858b3f from peer 9e4fa889fab047baa5bd3dcd77b2ca8f at {username='slave'} at 127.9.176.198:41673: session id = 9e4fa889fab047baa5bd3dcd77b2ca8f-930645182ab748d7b864fece1e858b3f
I20240502 01:53:32.029327 13624 tablet_copy_source_session.cc:205] T 930645182ab748d7b864fece1e858b3f P 8446b7a7ea67486288d174eea7a3b3c6: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:32.036788 15764 tablet_replica.cc:330] T d38a05f7ffa94cc5abc50a76a83692b5 P 5359629f948641f4b56e35ad6b6a15c4: stopping tablet replica
I20240502 01:53:32.039438 15760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 930645182ab748d7b864fece1e858b3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:32.046519 15760 tablet_copy_client.cc:783] T 930645182ab748d7b864fece1e858b3f P 9e4fa889fab047baa5bd3dcd77b2ca8f: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:32.046816 15760 tablet_copy_client.cc:647] T 930645182ab748d7b864fece1e858b3f P 9e4fa889fab047baa5bd3dcd77b2ca8f: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:32.051833 15764 raft_consensus.cc:2229] T d38a05f7ffa94cc5abc50a76a83692b5 P 5359629f948641f4b56e35ad6b6a15c4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:32.052256 15764 raft_consensus.cc:2258] T d38a05f7ffa94cc5abc50a76a83692b5 P 5359629f948641f4b56e35ad6b6a15c4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:32.055434 15764 ts_tablet_manager.cc:1875] T d38a05f7ffa94cc5abc50a76a83692b5 P 5359629f948641f4b56e35ad6b6a15c4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:53:32.056592 15751 tablet_copy_client.cc:523] T d4511dd1019c4bdcbf9fc7e4bfe2afb7 P 9e4fa889fab047baa5bd3dcd77b2ca8f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:32.061300 15751 tablet_bootstrap.cc:492] T d4511dd1019c4bdcbf9fc7e4bfe2afb7 P 9e4fa889fab047baa5bd3dcd77b2ca8f: Bootstrap starting.
I20240502 01:53:32.075269 15760 tablet_copy_client.cc:523] T 930645182ab748d7b864fece1e858b3f P 9e4fa889fab047baa5bd3dcd77b2ca8f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:32.079370 15760 tablet_bootstrap.cc:492] T 930645182ab748d7b864fece1e858b3f P 9e4fa889fab047baa5bd3dcd77b2ca8f: Bootstrap starting.
W20240502 01:53:32.120568 13671 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.176.199:38645, user_credentials={real_user=slave}} blocked reactor thread for 46318.3us
I20240502 01:53:32.179214 15764 ts_tablet_manager.cc:1888] T d38a05f7ffa94cc5abc50a76a83692b5 P 5359629f948641f4b56e35ad6b6a15c4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240502 01:53:32.179394 15764 log.cc:1196] T d38a05f7ffa94cc5abc50a76a83692b5 P 5359629f948641f4b56e35ad6b6a15c4: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/wal/wals/d38a05f7ffa94cc5abc50a76a83692b5
I20240502 01:53:32.206102 13313 catalog_manager.cc:4830] TS 5359629f948641f4b56e35ad6b6a15c4 (127.9.176.193:33849): tablet d38a05f7ffa94cc5abc50a76a83692b5 (table TestTable [id=821687150308482a8f601ee7300ef713]) successfully deleted
I20240502 01:53:32.257123 14954 raft_consensus.cc:1203] T b38a1fb4d1694c609a932e8816fc3514 P 7e74a3d5df6a483c95c9dabfc99577bc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.63->[1.64-1.81]   Dedup: 1.72->[1.73-1.81]
I20240502 01:53:32.318667 15476 raft_consensus.cc:1050] T b38a1fb4d1694c609a932e8816fc3514 P 5359629f948641f4b56e35ad6b6a15c4: attempting to promote NON_VOTER 7e74a3d5df6a483c95c9dabfc99577bc to VOTER
I20240502 01:53:32.319505 15476 consensus_queue.cc:237] T b38a1fb4d1694c609a932e8816fc3514 P 5359629f948641f4b56e35ad6b6a15c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 81, Committed index: 81, Last appended: 1.81, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: false } }
I20240502 01:53:32.322866 14952 raft_consensus.cc:1261] T b38a1fb4d1694c609a932e8816fc3514 P 7e74a3d5df6a483c95c9dabfc99577bc [term 1 LEARNER]: Refusing update from remote peer 5359629f948641f4b56e35ad6b6a15c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 81. Preceding OpId from leader: term: 1 index: 82. (index mismatch)
I20240502 01:53:32.322889 13735 raft_consensus.cc:1261] T b38a1fb4d1694c609a932e8816fc3514 P b4abc88ed2e54830ae3a317629916117 [term 1 FOLLOWER]: Refusing update from remote peer 5359629f948641f4b56e35ad6b6a15c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 81. Preceding OpId from leader: term: 1 index: 82. (index mismatch)
I20240502 01:53:32.323611 15643 consensus_queue.cc:1035] T b38a1fb4d1694c609a932e8816fc3514 P 5359629f948641f4b56e35ad6b6a15c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 82, Last known committed idx: 81, Time since last communication: 0.000s
I20240502 01:53:32.324851 15356 consensus_queue.cc:1035] T b38a1fb4d1694c609a932e8816fc3514 P 5359629f948641f4b56e35ad6b6a15c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 82, Last known committed idx: 81, Time since last communication: 0.000s
I20240502 01:53:32.328295 15356 raft_consensus.cc:2932] T b38a1fb4d1694c609a932e8816fc3514 P 5359629f948641f4b56e35ad6b6a15c4 [term 1 LEADER]: Committing config change with OpId 1.82: config changed from index 64 to 82, 7e74a3d5df6a483c95c9dabfc99577bc (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: false } } }
I20240502 01:53:32.330452 14952 raft_consensus.cc:2932] T b38a1fb4d1694c609a932e8816fc3514 P 7e74a3d5df6a483c95c9dabfc99577bc [term 1 FOLLOWER]: Committing config change with OpId 1.82: config changed from index 64 to 82, 7e74a3d5df6a483c95c9dabfc99577bc (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: false } } }
I20240502 01:53:32.332356 13735 raft_consensus.cc:2932] T b38a1fb4d1694c609a932e8816fc3514 P b4abc88ed2e54830ae3a317629916117 [term 1 FOLLOWER]: Committing config change with OpId 1.82: config changed from index 64 to 82, 7e74a3d5df6a483c95c9dabfc99577bc (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: false } } }
I20240502 01:53:32.340234 13326 catalog_manager.cc:5484] T b38a1fb4d1694c609a932e8816fc3514 P 7e74a3d5df6a483c95c9dabfc99577bc reported cstate change: config changed from index 64 to 82, 7e74a3d5df6a483c95c9dabfc99577bc (127.9.176.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5359629f948641f4b56e35ad6b6a15c4" committed_config { opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: false } } }
I20240502 01:53:32.355463 13470 consensus_queue.cc:237] T b38a1fb4d1694c609a932e8816fc3514 P 5359629f948641f4b56e35ad6b6a15c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 83, Committed index: 83, Last appended: 1.83, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: false } }
I20240502 01:53:32.358042 14952 raft_consensus.cc:1261] T b38a1fb4d1694c609a932e8816fc3514 P 7e74a3d5df6a483c95c9dabfc99577bc [term 1 FOLLOWER]: Refusing update from remote peer 5359629f948641f4b56e35ad6b6a15c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 83. Preceding OpId from leader: term: 1 index: 84. (index mismatch)
I20240502 01:53:32.358654 15360 consensus_queue.cc:1035] T b38a1fb4d1694c609a932e8816fc3514 P 5359629f948641f4b56e35ad6b6a15c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 84, Last known committed idx: 83, Time since last communication: 0.000s
I20240502 01:53:32.361538 15643 raft_consensus.cc:2932] T b38a1fb4d1694c609a932e8816fc3514 P 5359629f948641f4b56e35ad6b6a15c4 [term 1 LEADER]: Committing config change with OpId 1.84: config changed from index 82 to 84, VOTER b4abc88ed2e54830ae3a317629916117 (127.9.176.195) evicted. New config: { opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: false } } }
I20240502 01:53:32.367377 13313 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b38a1fb4d1694c609a932e8816fc3514 with cas_config_opid_index 82: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:53:32.367522 14954 raft_consensus.cc:2932] T b38a1fb4d1694c609a932e8816fc3514 P 7e74a3d5df6a483c95c9dabfc99577bc [term 1 FOLLOWER]: Committing config change with OpId 1.84: config changed from index 82 to 84, VOTER b4abc88ed2e54830ae3a317629916117 (127.9.176.195) evicted. New config: { opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: false } } }
I20240502 01:53:32.371129 13326 catalog_manager.cc:5484] T b38a1fb4d1694c609a932e8816fc3514 P 5359629f948641f4b56e35ad6b6a15c4 reported cstate change: config changed from index 82 to 84, VOTER b4abc88ed2e54830ae3a317629916117 (127.9.176.195) evicted. New cstate: current_term: 1 leader_uuid: "5359629f948641f4b56e35ad6b6a15c4" committed_config { opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:53:32.393587 13715 tablet_service.cc:1514] Processing DeleteTablet for tablet b38a1fb4d1694c609a932e8816fc3514 with delete_type TABLET_DATA_TOMBSTONED (TS b4abc88ed2e54830ae3a317629916117 not found in new config with opid_index 84) from {username='slave'} at 127.0.0.1:44986
I20240502 01:53:32.476123 15789 tablet_replica.cc:330] T b38a1fb4d1694c609a932e8816fc3514 P b4abc88ed2e54830ae3a317629916117: stopping tablet replica
I20240502 01:53:32.482268 15789 raft_consensus.cc:2229] T b38a1fb4d1694c609a932e8816fc3514 P b4abc88ed2e54830ae3a317629916117 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:32.484304 15789 raft_consensus.cc:2258] T b38a1fb4d1694c609a932e8816fc3514 P b4abc88ed2e54830ae3a317629916117 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:32.531932 15789 ts_tablet_manager.cc:1875] T b38a1fb4d1694c609a932e8816fc3514 P b4abc88ed2e54830ae3a317629916117: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:53:32.555356 15789 ts_tablet_manager.cc:1888] T b38a1fb4d1694c609a932e8816fc3514 P b4abc88ed2e54830ae3a317629916117: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.82
I20240502 01:53:32.555665 15789 log.cc:1196] T b38a1fb4d1694c609a932e8816fc3514 P b4abc88ed2e54830ae3a317629916117: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/wal/wals/b38a1fb4d1694c609a932e8816fc3514
I20240502 01:53:32.560629 13312 catalog_manager.cc:4830] TS b4abc88ed2e54830ae3a317629916117 (127.9.176.195:41211): tablet b38a1fb4d1694c609a932e8816fc3514 (table rebalance_test_table_1 [id=8ebdfe08a0b244819813a774f9d721fc]) successfully deleted
I20240502 01:53:32.662622 15732 tablet_bootstrap.cc:492] T 01fb93c5dab3401fa5a103ead4dd9708 P 7505fefeaf804ed8aeb3672f386ab959: Bootstrap replayed 1/1 log segments. Stats: ops{read=94 overwritten=0 applied=94 ignored=0} inserts{seen=766 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:53:32.663237 15732 tablet_bootstrap.cc:492] T 01fb93c5dab3401fa5a103ead4dd9708 P 7505fefeaf804ed8aeb3672f386ab959: Bootstrap complete.
I20240502 01:53:32.663619 15732 ts_tablet_manager.cc:1367] T 01fb93c5dab3401fa5a103ead4dd9708 P 7505fefeaf804ed8aeb3672f386ab959: Time spent bootstrapping tablet: real 0.653s	user 0.158s	sys 0.025s
I20240502 01:53:32.664608 15732 raft_consensus.cc:348] T 01fb93c5dab3401fa5a103ead4dd9708 P 7505fefeaf804ed8aeb3672f386ab959 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 86 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } attrs { replace: true } } peers { permanent_uuid: "7505fefeaf804ed8aeb3672f386ab959" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 38645 } attrs { promote: true } }
I20240502 01:53:32.664872 15732 raft_consensus.cc:729] T 01fb93c5dab3401fa5a103ead4dd9708 P 7505fefeaf804ed8aeb3672f386ab959 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7505fefeaf804ed8aeb3672f386ab959, State: Initialized, Role: LEARNER
I20240502 01:53:32.665086 15732 consensus_queue.cc:260] T 01fb93c5dab3401fa5a103ead4dd9708 P 7505fefeaf804ed8aeb3672f386ab959 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 94, Last appended: 1.94, Last appended by leader: 94, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 86 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } attrs { replace: true } } peers { permanent_uuid: "7505fefeaf804ed8aeb3672f386ab959" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 38645 } attrs { promote: true } }
I20240502 01:53:32.678247 15732 ts_tablet_manager.cc:1398] T 01fb93c5dab3401fa5a103ead4dd9708 P 7505fefeaf804ed8aeb3672f386ab959: Time spent starting tablet: real 0.014s	user 0.004s	sys 0.004s
I20240502 01:53:32.679020 13491 tablet_copy_service.cc:342] P 5359629f948641f4b56e35ad6b6a15c4: Request end of tablet copy session 7505fefeaf804ed8aeb3672f386ab959-01fb93c5dab3401fa5a103ead4dd9708 received from {username='slave'} at 127.9.176.199:45881
I20240502 01:53:32.679162 13491 tablet_copy_service.cc:433] P 5359629f948641f4b56e35ad6b6a15c4: ending tablet copy session 7505fefeaf804ed8aeb3672f386ab959-01fb93c5dab3401fa5a103ead4dd9708 on tablet 01fb93c5dab3401fa5a103ead4dd9708 with peer 7505fefeaf804ed8aeb3672f386ab959
I20240502 01:53:32.870237 15760 tablet_bootstrap.cc:492] T 930645182ab748d7b864fece1e858b3f P 9e4fa889fab047baa5bd3dcd77b2ca8f: Bootstrap replayed 1/1 log segments. Stats: ops{read=75 overwritten=0 applied=75 ignored=0} inserts{seen=760 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:53:32.870736 15760 tablet_bootstrap.cc:492] T 930645182ab748d7b864fece1e858b3f P 9e4fa889fab047baa5bd3dcd77b2ca8f: Bootstrap complete.
I20240502 01:53:32.871017 15760 ts_tablet_manager.cc:1367] T 930645182ab748d7b864fece1e858b3f P 9e4fa889fab047baa5bd3dcd77b2ca8f: Time spent bootstrapping tablet: real 0.792s	user 0.135s	sys 0.013s
I20240502 01:53:32.883498 15751 tablet_bootstrap.cc:492] T d4511dd1019c4bdcbf9fc7e4bfe2afb7 P 9e4fa889fab047baa5bd3dcd77b2ca8f: Bootstrap replayed 1/1 log segments. Stats: ops{read=92 overwritten=0 applied=92 ignored=0} inserts{seen=723 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:53:32.884068 15751 tablet_bootstrap.cc:492] T d4511dd1019c4bdcbf9fc7e4bfe2afb7 P 9e4fa889fab047baa5bd3dcd77b2ca8f: Bootstrap complete.
I20240502 01:53:32.884387 15751 ts_tablet_manager.cc:1367] T d4511dd1019c4bdcbf9fc7e4bfe2afb7 P 9e4fa889fab047baa5bd3dcd77b2ca8f: Time spent bootstrapping tablet: real 0.823s	user 0.150s	sys 0.029s
I20240502 01:53:32.884260 15760 raft_consensus.cc:348] T 930645182ab748d7b864fece1e858b3f P 9e4fa889fab047baa5bd3dcd77b2ca8f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: true } }
I20240502 01:53:32.884518 15760 raft_consensus.cc:729] T 930645182ab748d7b864fece1e858b3f P 9e4fa889fab047baa5bd3dcd77b2ca8f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9e4fa889fab047baa5bd3dcd77b2ca8f, State: Initialized, Role: LEARNER
I20240502 01:53:32.884744 15760 consensus_queue.cc:260] T 930645182ab748d7b864fece1e858b3f P 9e4fa889fab047baa5bd3dcd77b2ca8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 75, Last appended: 1.75, Last appended by leader: 75, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: true } }
I20240502 01:53:32.885227 15751 raft_consensus.cc:348] T d4511dd1019c4bdcbf9fc7e4bfe2afb7 P 9e4fa889fab047baa5bd3dcd77b2ca8f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 88 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: true } }
I20240502 01:53:32.885571 15751 raft_consensus.cc:729] T d4511dd1019c4bdcbf9fc7e4bfe2afb7 P 9e4fa889fab047baa5bd3dcd77b2ca8f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9e4fa889fab047baa5bd3dcd77b2ca8f, State: Initialized, Role: LEARNER
I20240502 01:53:32.885833 15751 consensus_queue.cc:260] T d4511dd1019c4bdcbf9fc7e4bfe2afb7 P 9e4fa889fab047baa5bd3dcd77b2ca8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 92, Last appended: 1.92, Last appended by leader: 92, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 88 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: true } }
I20240502 01:53:32.976120 15760 ts_tablet_manager.cc:1398] T 930645182ab748d7b864fece1e858b3f P 9e4fa889fab047baa5bd3dcd77b2ca8f: Time spent starting tablet: real 0.105s	user 0.019s	sys 0.045s
I20240502 01:53:32.991837 13624 tablet_copy_service.cc:342] P 8446b7a7ea67486288d174eea7a3b3c6: Request end of tablet copy session 9e4fa889fab047baa5bd3dcd77b2ca8f-930645182ab748d7b864fece1e858b3f received from {username='slave'} at 127.9.176.198:41673
I20240502 01:53:32.992046 13624 tablet_copy_service.cc:433] P 8446b7a7ea67486288d174eea7a3b3c6: ending tablet copy session 9e4fa889fab047baa5bd3dcd77b2ca8f-930645182ab748d7b864fece1e858b3f on tablet 930645182ab748d7b864fece1e858b3f with peer 9e4fa889fab047baa5bd3dcd77b2ca8f
I20240502 01:53:32.993547 15751 ts_tablet_manager.cc:1398] T d4511dd1019c4bdcbf9fc7e4bfe2afb7 P 9e4fa889fab047baa5bd3dcd77b2ca8f: Time spent starting tablet: real 0.109s	user 0.024s	sys 0.028s
I20240502 01:53:32.996824 13624 tablet_copy_service.cc:342] P 8446b7a7ea67486288d174eea7a3b3c6: Request end of tablet copy session 9e4fa889fab047baa5bd3dcd77b2ca8f-d4511dd1019c4bdcbf9fc7e4bfe2afb7 received from {username='slave'} at 127.9.176.198:41673
I20240502 01:53:32.996996 13624 tablet_copy_service.cc:433] P 8446b7a7ea67486288d174eea7a3b3c6: ending tablet copy session 9e4fa889fab047baa5bd3dcd77b2ca8f-d4511dd1019c4bdcbf9fc7e4bfe2afb7 on tablet d4511dd1019c4bdcbf9fc7e4bfe2afb7 with peer 9e4fa889fab047baa5bd3dcd77b2ca8f
I20240502 01:53:33.167696 15226 raft_consensus.cc:1203] T 01fb93c5dab3401fa5a103ead4dd9708 P 7505fefeaf804ed8aeb3672f386ab959 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.85->[1.86-1.111]   Dedup: 1.94->[1.95-1.111]
I20240502 01:53:33.241653 15087 raft_consensus.cc:1203] T 930645182ab748d7b864fece1e858b3f P 9e4fa889fab047baa5bd3dcd77b2ca8f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.70->[1.71-1.85]   Dedup: 1.75->[1.76-1.85]
I20240502 01:53:33.266898 15085 raft_consensus.cc:1203] T d4511dd1019c4bdcbf9fc7e4bfe2afb7 P 9e4fa889fab047baa5bd3dcd77b2ca8f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.87->[1.88-1.109]   Dedup: 1.92->[1.93-1.109]
I20240502 01:53:33.452105 15847 raft_consensus.cc:1050] T d4511dd1019c4bdcbf9fc7e4bfe2afb7 P 8446b7a7ea67486288d174eea7a3b3c6: attempting to promote NON_VOTER 9e4fa889fab047baa5bd3dcd77b2ca8f to VOTER
I20240502 01:53:33.456162 15356 raft_consensus.cc:1050] T 01fb93c5dab3401fa5a103ead4dd9708 P 5359629f948641f4b56e35ad6b6a15c4: attempting to promote NON_VOTER 7505fefeaf804ed8aeb3672f386ab959 to VOTER
I20240502 01:53:33.456957 15356 consensus_queue.cc:237] T 01fb93c5dab3401fa5a103ead4dd9708 P 5359629f948641f4b56e35ad6b6a15c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 114, Committed index: 114, Last appended: 1.114, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 115 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } attrs { replace: true } } peers { permanent_uuid: "7505fefeaf804ed8aeb3672f386ab959" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 38645 } attrs { promote: false } }
I20240502 01:53:33.458412 15403 raft_consensus.cc:1050] T 930645182ab748d7b864fece1e858b3f P 8446b7a7ea67486288d174eea7a3b3c6: attempting to promote NON_VOTER 9e4fa889fab047baa5bd3dcd77b2ca8f to VOTER
I20240502 01:53:33.459713 15228 raft_consensus.cc:1261] T 01fb93c5dab3401fa5a103ead4dd9708 P 7505fefeaf804ed8aeb3672f386ab959 [term 1 LEARNER]: Refusing update from remote peer 5359629f948641f4b56e35ad6b6a15c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 114. Preceding OpId from leader: term: 1 index: 115. (index mismatch)
I20240502 01:53:33.460386 15476 consensus_queue.cc:1035] T 01fb93c5dab3401fa5a103ead4dd9708 P 5359629f948641f4b56e35ad6b6a15c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7505fefeaf804ed8aeb3672f386ab959" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 38645 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 115, Last known committed idx: 114, Time since last communication: 0.000s
I20240502 01:53:33.464007 15356 raft_consensus.cc:2932] T 01fb93c5dab3401fa5a103ead4dd9708 P 5359629f948641f4b56e35ad6b6a15c4 [term 1 LEADER]: Committing config change with OpId 1.115: config changed from index 86 to 115, 7505fefeaf804ed8aeb3672f386ab959 (127.9.176.199) changed from NON_VOTER to VOTER. New config: { opid_index: 115 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } attrs { replace: true } } peers { permanent_uuid: "7505fefeaf804ed8aeb3672f386ab959" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 38645 } attrs { promote: false } } }
I20240502 01:53:33.464856 15228 raft_consensus.cc:2932] T 01fb93c5dab3401fa5a103ead4dd9708 P 7505fefeaf804ed8aeb3672f386ab959 [term 1 FOLLOWER]: Committing config change with OpId 1.115: config changed from index 86 to 115, 7505fefeaf804ed8aeb3672f386ab959 (127.9.176.199) changed from NON_VOTER to VOTER. New config: { opid_index: 115 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } attrs { replace: true } } peers { permanent_uuid: "7505fefeaf804ed8aeb3672f386ab959" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 38645 } attrs { promote: false } } }
I20240502 01:53:33.469540 13598 raft_consensus.cc:1261] T 01fb93c5dab3401fa5a103ead4dd9708 P 8446b7a7ea67486288d174eea7a3b3c6 [term 1 FOLLOWER]: Refusing update from remote peer 5359629f948641f4b56e35ad6b6a15c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 114. Preceding OpId from leader: term: 1 index: 115. (index mismatch)
I20240502 01:53:33.471158 15817 consensus_queue.cc:1035] T 01fb93c5dab3401fa5a103ead4dd9708 P 5359629f948641f4b56e35ad6b6a15c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 115, Last known committed idx: 114, Time since last communication: 0.000s
I20240502 01:53:33.471148 13326 catalog_manager.cc:5484] T 01fb93c5dab3401fa5a103ead4dd9708 P 7505fefeaf804ed8aeb3672f386ab959 reported cstate change: config changed from index 86 to 115, 7505fefeaf804ed8aeb3672f386ab959 (127.9.176.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5359629f948641f4b56e35ad6b6a15c4" committed_config { opid_index: 115 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } attrs { replace: true } } peers { permanent_uuid: "7505fefeaf804ed8aeb3672f386ab959" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 38645 } attrs { promote: false } } }
I20240502 01:53:33.477677 13599 raft_consensus.cc:2932] T 01fb93c5dab3401fa5a103ead4dd9708 P 8446b7a7ea67486288d174eea7a3b3c6 [term 1 FOLLOWER]: Committing config change with OpId 1.115: config changed from index 86 to 115, 7505fefeaf804ed8aeb3672f386ab959 (127.9.176.199) changed from NON_VOTER to VOTER. New config: { opid_index: 115 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } attrs { replace: true } } peers { permanent_uuid: "7505fefeaf804ed8aeb3672f386ab959" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 38645 } attrs { promote: false } } }
I20240502 01:53:33.543678 13470 consensus_queue.cc:237] T 01fb93c5dab3401fa5a103ead4dd9708 P 5359629f948641f4b56e35ad6b6a15c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 115, Committed index: 115, Last appended: 1.115, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 116 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "7505fefeaf804ed8aeb3672f386ab959" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 38645 } attrs { promote: false } }
I20240502 01:53:33.546396 15228 raft_consensus.cc:1261] T 01fb93c5dab3401fa5a103ead4dd9708 P 7505fefeaf804ed8aeb3672f386ab959 [term 1 FOLLOWER]: Refusing update from remote peer 5359629f948641f4b56e35ad6b6a15c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 115. Preceding OpId from leader: term: 1 index: 116. (index mismatch)
I20240502 01:53:33.547386 15360 consensus_queue.cc:1035] T 01fb93c5dab3401fa5a103ead4dd9708 P 5359629f948641f4b56e35ad6b6a15c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7505fefeaf804ed8aeb3672f386ab959" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 38645 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 116, Last known committed idx: 115, Time since last communication: 0.001s
I20240502 01:53:33.552270 15817 raft_consensus.cc:2932] T 01fb93c5dab3401fa5a103ead4dd9708 P 5359629f948641f4b56e35ad6b6a15c4 [term 1 LEADER]: Committing config change with OpId 1.116: config changed from index 115 to 116, VOTER 8446b7a7ea67486288d174eea7a3b3c6 (127.9.176.194) evicted. New config: { opid_index: 116 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "7505fefeaf804ed8aeb3672f386ab959" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 38645 } attrs { promote: false } } }
I20240502 01:53:33.563316 13318 catalog_manager.cc:5484] T 01fb93c5dab3401fa5a103ead4dd9708 P 5359629f948641f4b56e35ad6b6a15c4 reported cstate change: config changed from index 115 to 116, VOTER 8446b7a7ea67486288d174eea7a3b3c6 (127.9.176.194) evicted. New cstate: current_term: 1 leader_uuid: "5359629f948641f4b56e35ad6b6a15c4" committed_config { opid_index: 116 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7505fefeaf804ed8aeb3672f386ab959" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 38645 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:53:33.563650 15228 raft_consensus.cc:2932] T 01fb93c5dab3401fa5a103ead4dd9708 P 7505fefeaf804ed8aeb3672f386ab959 [term 1 FOLLOWER]: Committing config change with OpId 1.116: config changed from index 115 to 116, VOTER 8446b7a7ea67486288d174eea7a3b3c6 (127.9.176.194) evicted. New config: { opid_index: 116 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "7505fefeaf804ed8aeb3672f386ab959" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 38645 } attrs { promote: false } } }
I20240502 01:53:33.569758 13584 tablet_service.cc:1514] Processing DeleteTablet for tablet 01fb93c5dab3401fa5a103ead4dd9708 with delete_type TABLET_DATA_TOMBSTONED (TS 8446b7a7ea67486288d174eea7a3b3c6 not found in new config with opid_index 116) from {username='slave'} at 127.0.0.1:41580
I20240502 01:53:33.575321 13313 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 01fb93c5dab3401fa5a103ead4dd9708 with cas_config_opid_index 115: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:53:33.586809 15857 tablet_replica.cc:330] T 01fb93c5dab3401fa5a103ead4dd9708 P 8446b7a7ea67486288d174eea7a3b3c6: stopping tablet replica
I20240502 01:53:33.633213 15857 raft_consensus.cc:2229] T 01fb93c5dab3401fa5a103ead4dd9708 P 8446b7a7ea67486288d174eea7a3b3c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:33.633462 15403 consensus_queue.cc:237] T 930645182ab748d7b864fece1e858b3f P 8446b7a7ea67486288d174eea7a3b3c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 86, Committed index: 86, Last appended: 1.87, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 88 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false } }
I20240502 01:53:33.634267 15857 raft_consensus.cc:2258] T 01fb93c5dab3401fa5a103ead4dd9708 P 8446b7a7ea67486288d174eea7a3b3c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:33.635002 15854 raft_consensus.cc:1013] T 930645182ab748d7b864fece1e858b3f P 8446b7a7ea67486288d174eea7a3b3c6 [term 1 LEADER]: attempt to promote peer 9e4fa889fab047baa5bd3dcd77b2ca8f: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 01:53:33.637099 13735 raft_consensus.cc:1261] T 930645182ab748d7b864fece1e858b3f P b4abc88ed2e54830ae3a317629916117 [term 1 FOLLOWER]: Refusing update from remote peer 8446b7a7ea67486288d174eea7a3b3c6: Log matching property violated. Preceding OpId in replica: term: 1 index: 86. Preceding OpId from leader: term: 1 index: 88. (index mismatch)
I20240502 01:53:33.649516 15857 ts_tablet_manager.cc:1875] T 01fb93c5dab3401fa5a103ead4dd9708 P 8446b7a7ea67486288d174eea7a3b3c6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240502 01:53:33.680815 13525 kernel_stack_watchdog.cc:198] Thread 13539 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:308 for 118ms:
Kernel stack:
(could not read kernel stack)
User stack:
    @     0x7f818060a980  (unknown) at ??:0
    @           0x90ef77  __sanitizer::internal_sched_yield() at ??:?
    @           0x87462d  __interceptor_pthread_create at ??:?
    @     0x7f817d278d11  kudu::Thread::StartThread() at ??:0
    @           0xa4d187  kudu::Thread::Create() at ??:?
    @     0x7f817d2997ab  kudu::ThreadPool::CreateThread() at ??:0
    @     0x7f817d2953ad  kudu::ThreadPool::DoSubmit() at ??:0
    @     0x7f817d29384f  kudu::ThreadPoolToken::Submit() at ??:0
    @     0x7f81889c69e8  kudu::consensus::Peer::ProcessResponse() at ??:0
    @     0x7f81889cd061  kudu::consensus::Peer::SendNextRequest()::$_3::operator()() at ??:0
    @     0x7f81889cce6e  std::_Function_handler<>::_M_invoke() at ??:0
    @           0xa60bcd  std::function<>::operator()() at ??:?
    @     0x7f8183eca8cf  kudu::rpc::Proxy::AsyncRequest()::$_1::operator()() at ??:0
    @     0x7f8183eca4ae  std::_Function_handler<>::_M_invoke() at ??:0
    @           0xa60bcd  std::function<>::operator()() at ??:?
    @     0x7f8183eb64e0  kudu::rpc::OutboundCall::CallCallback() at ??:0
I20240502 01:53:33.696336 15449 consensus_queue.cc:1035] T 930645182ab748d7b864fece1e858b3f P 8446b7a7ea67486288d174eea7a3b3c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 88, Last known committed idx: 86, Time since last communication: 0.000s
I20240502 01:53:33.700073 13734 raft_consensus.cc:1203] T 930645182ab748d7b864fece1e858b3f P b4abc88ed2e54830ae3a317629916117 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.86->[1.87-1.88]   Dedup: 1.87->[1.88-1.88]
W20240502 01:53:33.703343 13539 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.176.198:38527, user_credentials={real_user=slave}} blocked reactor thread for 251302us
I20240502 01:53:33.704313 15847 consensus_queue.cc:237] T d4511dd1019c4bdcbf9fc7e4bfe2afb7 P 8446b7a7ea67486288d174eea7a3b3c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 112, Committed index: 112, Last appended: 1.113, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 114 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false } }
I20240502 01:53:33.705127 15085 raft_consensus.cc:1261] T 930645182ab748d7b864fece1e858b3f P 9e4fa889fab047baa5bd3dcd77b2ca8f [term 1 LEARNER]: Refusing update from remote peer 8446b7a7ea67486288d174eea7a3b3c6: Log matching property violated. Preceding OpId in replica: term: 1 index: 86. Preceding OpId from leader: term: 1 index: 88. (index mismatch)
I20240502 01:53:33.706847 15851 raft_consensus.cc:1013] T d4511dd1019c4bdcbf9fc7e4bfe2afb7 P 8446b7a7ea67486288d174eea7a3b3c6 [term 1 LEADER]: attempt to promote peer 9e4fa889fab047baa5bd3dcd77b2ca8f: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 01:53:33.708190 15085 raft_consensus.cc:1261] T d4511dd1019c4bdcbf9fc7e4bfe2afb7 P 9e4fa889fab047baa5bd3dcd77b2ca8f [term 1 LEARNER]: Refusing update from remote peer 8446b7a7ea67486288d174eea7a3b3c6: Log matching property violated. Preceding OpId in replica: term: 1 index: 113. Preceding OpId from leader: term: 1 index: 114. (index mismatch)
I20240502 01:53:33.708416 15498 consensus_queue.cc:1035] T 930645182ab748d7b864fece1e858b3f P 8446b7a7ea67486288d174eea7a3b3c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 88, Last known committed idx: 86, Time since last communication: 0.000s
I20240502 01:53:33.710434 15851 raft_consensus.cc:2932] T 930645182ab748d7b864fece1e858b3f P 8446b7a7ea67486288d174eea7a3b3c6 [term 1 LEADER]: Committing config change with OpId 1.88: config changed from index 71 to 88, 9e4fa889fab047baa5bd3dcd77b2ca8f (127.9.176.198) changed from NON_VOTER to VOTER. New config: { opid_index: 88 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false } } }
I20240502 01:53:33.712493 15857 ts_tablet_manager.cc:1888] T 01fb93c5dab3401fa5a103ead4dd9708 P 8446b7a7ea67486288d174eea7a3b3c6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.115
I20240502 01:53:33.712699 15857 log.cc:1196] T 01fb93c5dab3401fa5a103ead4dd9708 P 8446b7a7ea67486288d174eea7a3b3c6: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/wal/wals/01fb93c5dab3401fa5a103ead4dd9708
I20240502 01:53:33.717411 13310 catalog_manager.cc:4830] TS 8446b7a7ea67486288d174eea7a3b3c6 (127.9.176.194:34851): tablet 01fb93c5dab3401fa5a103ead4dd9708 (table rebalance_test_table_0 [id=42b769e9abfb4c36b2891f39a41750d2]) successfully deleted
I20240502 01:53:33.717491 13326 catalog_manager.cc:5484] T 930645182ab748d7b864fece1e858b3f P 8446b7a7ea67486288d174eea7a3b3c6 reported cstate change: config changed from index 71 to 88, 9e4fa889fab047baa5bd3dcd77b2ca8f (127.9.176.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8446b7a7ea67486288d174eea7a3b3c6" committed_config { opid_index: 88 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:33.719930 13734 raft_consensus.cc:1261] T d4511dd1019c4bdcbf9fc7e4bfe2afb7 P b4abc88ed2e54830ae3a317629916117 [term 1 FOLLOWER]: Refusing update from remote peer 8446b7a7ea67486288d174eea7a3b3c6: Log matching property violated. Preceding OpId in replica: term: 1 index: 113. Preceding OpId from leader: term: 1 index: 114. (index mismatch)
I20240502 01:53:33.725515 13732 raft_consensus.cc:2932] T 930645182ab748d7b864fece1e858b3f P b4abc88ed2e54830ae3a317629916117 [term 1 FOLLOWER]: Committing config change with OpId 1.88: config changed from index 71 to 88, 9e4fa889fab047baa5bd3dcd77b2ca8f (127.9.176.198) changed from NON_VOTER to VOTER. New config: { opid_index: 88 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false } } }
I20240502 01:53:33.730083 15449 consensus_queue.cc:1035] T d4511dd1019c4bdcbf9fc7e4bfe2afb7 P 8446b7a7ea67486288d174eea7a3b3c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 114, Last known committed idx: 113, Time since last communication: 0.000s
I20240502 01:53:33.732818 15084 raft_consensus.cc:2932] T 930645182ab748d7b864fece1e858b3f P 9e4fa889fab047baa5bd3dcd77b2ca8f [term 1 FOLLOWER]: Committing config change with OpId 1.88: config changed from index 71 to 88, 9e4fa889fab047baa5bd3dcd77b2ca8f (127.9.176.198) changed from NON_VOTER to VOTER. New config: { opid_index: 88 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false } } }
I20240502 01:53:33.741595 13599 consensus_queue.cc:237] T 930645182ab748d7b864fece1e858b3f P 8446b7a7ea67486288d174eea7a3b3c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 88, Committed index: 88, Last appended: 1.88, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 89 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false } }
I20240502 01:53:33.746141 15085 raft_consensus.cc:1261] T 930645182ab748d7b864fece1e858b3f P 9e4fa889fab047baa5bd3dcd77b2ca8f [term 1 FOLLOWER]: Refusing update from remote peer 8446b7a7ea67486288d174eea7a3b3c6: Log matching property violated. Preceding OpId in replica: term: 1 index: 88. Preceding OpId from leader: term: 1 index: 89. (index mismatch)
I20240502 01:53:33.745965 15781 raft_consensus.cc:2932] T d4511dd1019c4bdcbf9fc7e4bfe2afb7 P 8446b7a7ea67486288d174eea7a3b3c6 [term 1 LEADER]: Committing config change with OpId 1.114: config changed from index 88 to 114, 9e4fa889fab047baa5bd3dcd77b2ca8f (127.9.176.198) changed from NON_VOTER to VOTER. New config: { opid_index: 114 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false } } }
I20240502 01:53:33.747798 15085 raft_consensus.cc:2932] T d4511dd1019c4bdcbf9fc7e4bfe2afb7 P 9e4fa889fab047baa5bd3dcd77b2ca8f [term 1 FOLLOWER]: Committing config change with OpId 1.114: config changed from index 88 to 114, 9e4fa889fab047baa5bd3dcd77b2ca8f (127.9.176.198) changed from NON_VOTER to VOTER. New config: { opid_index: 114 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false } } }
I20240502 01:53:33.748592 15847 consensus_queue.cc:1035] T d4511dd1019c4bdcbf9fc7e4bfe2afb7 P 8446b7a7ea67486288d174eea7a3b3c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 114, Last known committed idx: 113, Time since last communication: 0.000s
I20240502 01:53:33.772254 13732 raft_consensus.cc:2932] T d4511dd1019c4bdcbf9fc7e4bfe2afb7 P b4abc88ed2e54830ae3a317629916117 [term 1 FOLLOWER]: Committing config change with OpId 1.114: config changed from index 88 to 114, 9e4fa889fab047baa5bd3dcd77b2ca8f (127.9.176.198) changed from NON_VOTER to VOTER. New config: { opid_index: 114 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false } } }
I20240502 01:53:33.780227 15847 consensus_queue.cc:1035] T 930645182ab748d7b864fece1e858b3f P 8446b7a7ea67486288d174eea7a3b3c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 89, Last known committed idx: 88, Time since last communication: 0.000s
I20240502 01:53:33.787209 13326 catalog_manager.cc:5484] T d4511dd1019c4bdcbf9fc7e4bfe2afb7 P 8446b7a7ea67486288d174eea7a3b3c6 reported cstate change: config changed from index 88 to 114, 9e4fa889fab047baa5bd3dcd77b2ca8f (127.9.176.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8446b7a7ea67486288d174eea7a3b3c6" committed_config { opid_index: 114 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:53:33.795557 13598 consensus_queue.cc:237] T d4511dd1019c4bdcbf9fc7e4bfe2afb7 P 8446b7a7ea67486288d174eea7a3b3c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 114, Committed index: 114, Last appended: 1.115, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 116 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false } }
W20240502 01:53:33.889729 13539 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.176.199:38645, user_credentials={real_user=slave}} blocked reactor thread for 97550.6us
I20240502 01:53:33.890645 15849 raft_consensus.cc:2932] T 930645182ab748d7b864fece1e858b3f P 8446b7a7ea67486288d174eea7a3b3c6 [term 1 LEADER]: Committing config change with OpId 1.89: config changed from index 88 to 89, VOTER b4abc88ed2e54830ae3a317629916117 (127.9.176.195) evicted. New config: { opid_index: 89 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false } } }
I20240502 01:53:33.894779 13310 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 930645182ab748d7b864fece1e858b3f with cas_config_opid_index 88: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:53:33.897169 13326 catalog_manager.cc:5484] T 930645182ab748d7b864fece1e858b3f P 8446b7a7ea67486288d174eea7a3b3c6 reported cstate change: config changed from index 88 to 89, VOTER b4abc88ed2e54830ae3a317629916117 (127.9.176.195) evicted. New cstate: current_term: 1 leader_uuid: "8446b7a7ea67486288d174eea7a3b3c6" committed_config { opid_index: 89 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:53:33.903064 13715 tablet_service.cc:1514] Processing DeleteTablet for tablet 930645182ab748d7b864fece1e858b3f with delete_type TABLET_DATA_TOMBSTONED (TS b4abc88ed2e54830ae3a317629916117 not found in new config with opid_index 89) from {username='slave'} at 127.0.0.1:44986
I20240502 01:53:33.903861 15086 raft_consensus.cc:1261] T d4511dd1019c4bdcbf9fc7e4bfe2afb7 P 9e4fa889fab047baa5bd3dcd77b2ca8f [term 1 FOLLOWER]: Refusing update from remote peer 8446b7a7ea67486288d174eea7a3b3c6: Log matching property violated. Preceding OpId in replica: term: 1 index: 114. Preceding OpId from leader: term: 1 index: 116. (index mismatch)
I20240502 01:53:33.904659 15083 raft_consensus.cc:2932] T 930645182ab748d7b864fece1e858b3f P 9e4fa889fab047baa5bd3dcd77b2ca8f [term 1 FOLLOWER]: Committing config change with OpId 1.89: config changed from index 88 to 89, VOTER b4abc88ed2e54830ae3a317629916117 (127.9.176.195) evicted. New config: { opid_index: 89 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false } } }
I20240502 01:53:33.916441 15498 consensus_queue.cc:1035] T d4511dd1019c4bdcbf9fc7e4bfe2afb7 P 8446b7a7ea67486288d174eea7a3b3c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 116, Last known committed idx: 114, Time since last communication: 0.000s
I20240502 01:53:33.943655 15849 raft_consensus.cc:2932] T d4511dd1019c4bdcbf9fc7e4bfe2afb7 P 8446b7a7ea67486288d174eea7a3b3c6 [term 1 LEADER]: Committing config change with OpId 1.116: config changed from index 114 to 116, VOTER b4abc88ed2e54830ae3a317629916117 (127.9.176.195) evicted. New config: { opid_index: 116 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false } } }
I20240502 01:53:33.949491 13310 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d4511dd1019c4bdcbf9fc7e4bfe2afb7 with cas_config_opid_index 114: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:53:33.950187 15868 tablet_replica.cc:330] T 930645182ab748d7b864fece1e858b3f P b4abc88ed2e54830ae3a317629916117: stopping tablet replica
I20240502 01:53:33.950402 15868 raft_consensus.cc:2229] T 930645182ab748d7b864fece1e858b3f P b4abc88ed2e54830ae3a317629916117 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:33.950690 15868 raft_consensus.cc:2258] T 930645182ab748d7b864fece1e858b3f P b4abc88ed2e54830ae3a317629916117 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:33.950973 13326 catalog_manager.cc:5484] T d4511dd1019c4bdcbf9fc7e4bfe2afb7 P 8446b7a7ea67486288d174eea7a3b3c6 reported cstate change: config changed from index 114 to 116, VOTER b4abc88ed2e54830ae3a317629916117 (127.9.176.195) evicted. New cstate: current_term: 1 leader_uuid: "8446b7a7ea67486288d174eea7a3b3c6" committed_config { opid_index: 116 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:53:33.965116 15083 raft_consensus.cc:2932] T d4511dd1019c4bdcbf9fc7e4bfe2afb7 P 9e4fa889fab047baa5bd3dcd77b2ca8f [term 1 FOLLOWER]: Committing config change with OpId 1.116: config changed from index 114 to 116, VOTER b4abc88ed2e54830ae3a317629916117 (127.9.176.195) evicted. New config: { opid_index: 116 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false } } }
I20240502 01:53:33.955507 13715 tablet_service.cc:1514] Processing DeleteTablet for tablet d4511dd1019c4bdcbf9fc7e4bfe2afb7 with delete_type TABLET_DATA_TOMBSTONED (TS b4abc88ed2e54830ae3a317629916117 not found in new config with opid_index 116) from {username='slave'} at 127.0.0.1:44986
I20240502 01:53:33.979700 15868 ts_tablet_manager.cc:1875] T 930645182ab748d7b864fece1e858b3f P b4abc88ed2e54830ae3a317629916117: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:53:33.999456 15868 ts_tablet_manager.cc:1888] T 930645182ab748d7b864fece1e858b3f P b4abc88ed2e54830ae3a317629916117: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.88
I20240502 01:53:33.999624 15868 log.cc:1196] T 930645182ab748d7b864fece1e858b3f P b4abc88ed2e54830ae3a317629916117: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/wal/wals/930645182ab748d7b864fece1e858b3f
I20240502 01:53:34.005187 15868 tablet_replica.cc:330] T d4511dd1019c4bdcbf9fc7e4bfe2afb7 P b4abc88ed2e54830ae3a317629916117: stopping tablet replica
I20240502 01:53:34.005383 15868 raft_consensus.cc:2229] T d4511dd1019c4bdcbf9fc7e4bfe2afb7 P b4abc88ed2e54830ae3a317629916117 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:34.005586 15868 raft_consensus.cc:2258] T d4511dd1019c4bdcbf9fc7e4bfe2afb7 P b4abc88ed2e54830ae3a317629916117 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:34.006601 15868 ts_tablet_manager.cc:1875] T d4511dd1019c4bdcbf9fc7e4bfe2afb7 P b4abc88ed2e54830ae3a317629916117: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:53:34.013844 13312 catalog_manager.cc:4830] TS b4abc88ed2e54830ae3a317629916117 (127.9.176.195:41211): tablet 930645182ab748d7b864fece1e858b3f (table rebalance_test_table_4 [id=7c97435ee22d42aaa3452b6e2abdaf75]) successfully deleted
I20240502 01:53:34.034193 15868 ts_tablet_manager.cc:1888] T d4511dd1019c4bdcbf9fc7e4bfe2afb7 P b4abc88ed2e54830ae3a317629916117: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.114
I20240502 01:53:34.034333 15868 log.cc:1196] T d4511dd1019c4bdcbf9fc7e4bfe2afb7 P b4abc88ed2e54830ae3a317629916117: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/wal/wals/d4511dd1019c4bdcbf9fc7e4bfe2afb7
I20240502 01:53:34.037904 13312 catalog_manager.cc:4830] TS b4abc88ed2e54830ae3a317629916117 (127.9.176.195:41211): tablet d4511dd1019c4bdcbf9fc7e4bfe2afb7 (table rebalance_test_table_0 [id=42b769e9abfb4c36b2891f39a41750d2]) successfully deleted
I20240502 01:53:34.562804 13470 consensus_queue.cc:237] T 71b0f9fbee82464994252126b2d631bc P 5359629f948641f4b56e35ad6b6a15c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 98, Committed index: 98, Last appended: 1.98, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } attrs { replace: true } } peers { permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: true } }
I20240502 01:53:34.565023 13599 raft_consensus.cc:1261] T 71b0f9fbee82464994252126b2d631bc P 8446b7a7ea67486288d174eea7a3b3c6 [term 1 FOLLOWER]: Refusing update from remote peer 5359629f948641f4b56e35ad6b6a15c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 98. Preceding OpId from leader: term: 1 index: 99. (index mismatch)
I20240502 01:53:34.565970 15360 consensus_queue.cc:1035] T 71b0f9fbee82464994252126b2d631bc P 5359629f948641f4b56e35ad6b6a15c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 99, Last known committed idx: 98, Time since last communication: 0.000s
W20240502 01:53:34.569299 13402 consensus_peers.cc:485] T 71b0f9fbee82464994252126b2d631bc P 5359629f948641f4b56e35ad6b6a15c4 -> Peer 7e74a3d5df6a483c95c9dabfc99577bc (127.9.176.197:35757): Couldn't send request to peer 7e74a3d5df6a483c95c9dabfc99577bc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 71b0f9fbee82464994252126b2d631bc. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:34.569494 15476 raft_consensus.cc:2932] T 71b0f9fbee82464994252126b2d631bc P 5359629f948641f4b56e35ad6b6a15c4 [term 1 LEADER]: Committing config change with OpId 1.99: config changed from index -1 to 99, NON_VOTER 7e74a3d5df6a483c95c9dabfc99577bc (127.9.176.197) added. New config: { opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } attrs { replace: true } } peers { permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: true } } }
I20240502 01:53:34.572216 13598 raft_consensus.cc:2932] T 71b0f9fbee82464994252126b2d631bc P 8446b7a7ea67486288d174eea7a3b3c6 [term 1 FOLLOWER]: Committing config change with OpId 1.99: config changed from index -1 to 99, NON_VOTER 7e74a3d5df6a483c95c9dabfc99577bc (127.9.176.197) added. New config: { opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } attrs { replace: true } } peers { permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: true } } }
I20240502 01:53:34.577306 13326 catalog_manager.cc:5484] T 71b0f9fbee82464994252126b2d631bc P 5359629f948641f4b56e35ad6b6a15c4 reported cstate change: config changed from index -1 to 99, NON_VOTER 7e74a3d5df6a483c95c9dabfc99577bc (127.9.176.197) added. New cstate: current_term: 1 leader_uuid: "5359629f948641f4b56e35ad6b6a15c4" committed_config { opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:35.166483 15948 ts_tablet_manager.cc:902] T 71b0f9fbee82464994252126b2d631bc P 7e74a3d5df6a483c95c9dabfc99577bc: Initiating tablet copy from peer 5359629f948641f4b56e35ad6b6a15c4 (127.9.176.193:33849)
I20240502 01:53:35.191951 15948 tablet_copy_client.cc:308] T 71b0f9fbee82464994252126b2d631bc P 7e74a3d5df6a483c95c9dabfc99577bc: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.193:33849
I20240502 01:53:35.196010 13491 tablet_copy_service.cc:140] P 5359629f948641f4b56e35ad6b6a15c4: Received BeginTabletCopySession request for tablet 71b0f9fbee82464994252126b2d631bc from peer 7e74a3d5df6a483c95c9dabfc99577bc ({username='slave'} at 127.9.176.197:57335)
I20240502 01:53:35.196235 13491 tablet_copy_service.cc:161] P 5359629f948641f4b56e35ad6b6a15c4: Beginning new tablet copy session on tablet 71b0f9fbee82464994252126b2d631bc from peer 7e74a3d5df6a483c95c9dabfc99577bc at {username='slave'} at 127.9.176.197:57335: session id = 7e74a3d5df6a483c95c9dabfc99577bc-71b0f9fbee82464994252126b2d631bc
I20240502 01:53:35.199836 13491 tablet_copy_source_session.cc:205] T 71b0f9fbee82464994252126b2d631bc P 5359629f948641f4b56e35ad6b6a15c4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:35.208081 15948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71b0f9fbee82464994252126b2d631bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:35.218479 15948 tablet_copy_client.cc:783] T 71b0f9fbee82464994252126b2d631bc P 7e74a3d5df6a483c95c9dabfc99577bc: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:35.222913 15948 tablet_copy_client.cc:647] T 71b0f9fbee82464994252126b2d631bc P 7e74a3d5df6a483c95c9dabfc99577bc: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:35.240934 15948 tablet_copy_client.cc:523] T 71b0f9fbee82464994252126b2d631bc P 7e74a3d5df6a483c95c9dabfc99577bc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:35.246245 15948 tablet_bootstrap.cc:492] T 71b0f9fbee82464994252126b2d631bc P 7e74a3d5df6a483c95c9dabfc99577bc: Bootstrap starting.
I20240502 01:53:35.568807 15948 tablet_bootstrap.cc:492] T 71b0f9fbee82464994252126b2d631bc P 7e74a3d5df6a483c95c9dabfc99577bc: Bootstrap replayed 1/1 log segments. Stats: ops{read=112 overwritten=0 applied=111 ignored=0} inserts{seen=783 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 01:53:35.569443 15948 tablet_bootstrap.cc:492] T 71b0f9fbee82464994252126b2d631bc P 7e74a3d5df6a483c95c9dabfc99577bc: Bootstrap complete.
I20240502 01:53:35.569761 15948 ts_tablet_manager.cc:1367] T 71b0f9fbee82464994252126b2d631bc P 7e74a3d5df6a483c95c9dabfc99577bc: Time spent bootstrapping tablet: real 0.324s	user 0.175s	sys 0.017s
I20240502 01:53:35.570613 15948 raft_consensus.cc:348] T 71b0f9fbee82464994252126b2d631bc P 7e74a3d5df6a483c95c9dabfc99577bc [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } attrs { replace: true } } peers { permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: true } }
I20240502 01:53:35.572052 15948 raft_consensus.cc:729] T 71b0f9fbee82464994252126b2d631bc P 7e74a3d5df6a483c95c9dabfc99577bc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7e74a3d5df6a483c95c9dabfc99577bc, State: Initialized, Role: LEARNER
I20240502 01:53:35.572291 15948 consensus_queue.cc:260] T 71b0f9fbee82464994252126b2d631bc P 7e74a3d5df6a483c95c9dabfc99577bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 111, Last appended: 1.112, Last appended by leader: 112, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } attrs { replace: true } } peers { permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: true } }
I20240502 01:53:35.573208 15948 ts_tablet_manager.cc:1398] T 71b0f9fbee82464994252126b2d631bc P 7e74a3d5df6a483c95c9dabfc99577bc: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.002s
I20240502 01:53:35.574582 13491 tablet_copy_service.cc:342] P 5359629f948641f4b56e35ad6b6a15c4: Request end of tablet copy session 7e74a3d5df6a483c95c9dabfc99577bc-71b0f9fbee82464994252126b2d631bc received from {username='slave'} at 127.9.176.197:57335
I20240502 01:53:35.574750 13491 tablet_copy_service.cc:433] P 5359629f948641f4b56e35ad6b6a15c4: ending tablet copy session 7e74a3d5df6a483c95c9dabfc99577bc-71b0f9fbee82464994252126b2d631bc on tablet 71b0f9fbee82464994252126b2d631bc with peer 7e74a3d5df6a483c95c9dabfc99577bc
I20240502 01:53:35.651571 14953 raft_consensus.cc:1203] T 71b0f9fbee82464994252126b2d631bc P 7e74a3d5df6a483c95c9dabfc99577bc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.98->[1.99-1.122]   Dedup: 1.112->[1.113-1.122]
I20240502 01:53:35.707593 15360 raft_consensus.cc:1050] T 71b0f9fbee82464994252126b2d631bc P 5359629f948641f4b56e35ad6b6a15c4: attempting to promote NON_VOTER 7e74a3d5df6a483c95c9dabfc99577bc to VOTER
I20240502 01:53:35.708427 15360 consensus_queue.cc:237] T 71b0f9fbee82464994252126b2d631bc P 5359629f948641f4b56e35ad6b6a15c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 124, Committed index: 124, Last appended: 1.124, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 125 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } attrs { replace: true } } peers { permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: false } }
I20240502 01:53:35.712550 14953 raft_consensus.cc:1261] T 71b0f9fbee82464994252126b2d631bc P 7e74a3d5df6a483c95c9dabfc99577bc [term 1 LEARNER]: Refusing update from remote peer 5359629f948641f4b56e35ad6b6a15c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 124. Preceding OpId from leader: term: 1 index: 126. (index mismatch)
I20240502 01:53:35.714731 13604 raft_consensus.cc:1261] T 71b0f9fbee82464994252126b2d631bc P 8446b7a7ea67486288d174eea7a3b3c6 [term 1 FOLLOWER]: Refusing update from remote peer 5359629f948641f4b56e35ad6b6a15c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 124. Preceding OpId from leader: term: 1 index: 126. (index mismatch)
I20240502 01:53:35.717628 15903 consensus_queue.cc:1035] T 71b0f9fbee82464994252126b2d631bc P 5359629f948641f4b56e35ad6b6a15c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 125, Last known committed idx: 124, Time since last communication: 0.000s
I20240502 01:53:35.718642 15903 consensus_queue.cc:1035] T 71b0f9fbee82464994252126b2d631bc P 5359629f948641f4b56e35ad6b6a15c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 125, Last known committed idx: 124, Time since last communication: 0.000s
I20240502 01:53:35.730798 15643 raft_consensus.cc:2932] T 71b0f9fbee82464994252126b2d631bc P 5359629f948641f4b56e35ad6b6a15c4 [term 1 LEADER]: Committing config change with OpId 1.125: config changed from index 99 to 125, 7e74a3d5df6a483c95c9dabfc99577bc (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 125 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } attrs { replace: true } } peers { permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: false } } }
I20240502 01:53:35.736023 13326 catalog_manager.cc:5484] T 71b0f9fbee82464994252126b2d631bc P 5359629f948641f4b56e35ad6b6a15c4 reported cstate change: config changed from index 99 to 125, 7e74a3d5df6a483c95c9dabfc99577bc (127.9.176.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5359629f948641f4b56e35ad6b6a15c4" committed_config { opid_index: 125 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:35.741891 13604 raft_consensus.cc:2932] T 71b0f9fbee82464994252126b2d631bc P 8446b7a7ea67486288d174eea7a3b3c6 [term 1 FOLLOWER]: Committing config change with OpId 1.125: config changed from index 99 to 125, 7e74a3d5df6a483c95c9dabfc99577bc (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 125 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } attrs { replace: true } } peers { permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: false } } }
I20240502 01:53:35.760073 14953 raft_consensus.cc:2932] T 71b0f9fbee82464994252126b2d631bc P 7e74a3d5df6a483c95c9dabfc99577bc [term 1 FOLLOWER]: Committing config change with OpId 1.125: config changed from index 99 to 125, 7e74a3d5df6a483c95c9dabfc99577bc (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 125 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } attrs { replace: true } } peers { permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: false } } }
I20240502 01:53:35.842113 13470 consensus_queue.cc:237] T 71b0f9fbee82464994252126b2d631bc P 5359629f948641f4b56e35ad6b6a15c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 127, Committed index: 127, Last appended: 1.128, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 129 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: false } }
I20240502 01:53:35.844705 14953 raft_consensus.cc:1261] T 71b0f9fbee82464994252126b2d631bc P 7e74a3d5df6a483c95c9dabfc99577bc [term 1 FOLLOWER]: Refusing update from remote peer 5359629f948641f4b56e35ad6b6a15c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 127. Preceding OpId from leader: term: 1 index: 129. (index mismatch)
I20240502 01:53:35.845391 15360 consensus_queue.cc:1035] T 71b0f9fbee82464994252126b2d631bc P 5359629f948641f4b56e35ad6b6a15c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 129, Last known committed idx: 127, Time since last communication: 0.000s
I20240502 01:53:35.855464 15903 raft_consensus.cc:2932] T 71b0f9fbee82464994252126b2d631bc P 5359629f948641f4b56e35ad6b6a15c4 [term 1 LEADER]: Committing config change with OpId 1.129: config changed from index 125 to 129, VOTER 8446b7a7ea67486288d174eea7a3b3c6 (127.9.176.194) evicted. New config: { opid_index: 129 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: false } } }
I20240502 01:53:35.864312 13313 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 71b0f9fbee82464994252126b2d631bc with cas_config_opid_index 125: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:53:35.869797 14953 raft_consensus.cc:2932] T 71b0f9fbee82464994252126b2d631bc P 7e74a3d5df6a483c95c9dabfc99577bc [term 1 FOLLOWER]: Committing config change with OpId 1.129: config changed from index 125 to 129, VOTER 8446b7a7ea67486288d174eea7a3b3c6 (127.9.176.194) evicted. New config: { opid_index: 129 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: false } } }
I20240502 01:53:35.873590 13325 catalog_manager.cc:5484] T 71b0f9fbee82464994252126b2d631bc P 5359629f948641f4b56e35ad6b6a15c4 reported cstate change: config changed from index 125 to 129, VOTER 8446b7a7ea67486288d174eea7a3b3c6 (127.9.176.194) evicted. New cstate: current_term: 1 leader_uuid: "5359629f948641f4b56e35ad6b6a15c4" committed_config { opid_index: 129 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:53:35.880445 13584 tablet_service.cc:1514] Processing DeleteTablet for tablet 71b0f9fbee82464994252126b2d631bc with delete_type TABLET_DATA_TOMBSTONED (TS 8446b7a7ea67486288d174eea7a3b3c6 not found in new config with opid_index 129) from {username='slave'} at 127.0.0.1:41580
I20240502 01:53:35.909427 15982 tablet_replica.cc:330] T 71b0f9fbee82464994252126b2d631bc P 8446b7a7ea67486288d174eea7a3b3c6: stopping tablet replica
I20240502 01:53:35.909723 15982 raft_consensus.cc:2229] T 71b0f9fbee82464994252126b2d631bc P 8446b7a7ea67486288d174eea7a3b3c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:35.909965 15982 pending_rounds.cc:62] T 71b0f9fbee82464994252126b2d631bc P 8446b7a7ea67486288d174eea7a3b3c6: Trying to abort 1 pending ops.
I20240502 01:53:35.910084 15982 pending_rounds.cc:68] T 71b0f9fbee82464994252126b2d631bc P 8446b7a7ea67486288d174eea7a3b3c6: Aborting op as it isn't in flight: id { term: 1 index: 128 } timestamp: 7023062285683884032 op_type: WRITE_OP write_request { tablet_id: "71b0f9fbee82464994252126b2d631bc" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\277W\333\014i\320\271\227\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""dd04f63de1849d01" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023062285639880704 authz_token { token_data: "\010\301\346\313\261\006\"3\n\005slave\022*\n 7c97435ee22d42aaa3452b6e2abdaf75\020\001\030\001 \001(\001" signature: "<redacted>""\010\214\207\364\030\021/jBd\274<\352\322R\362o\203U\324\373\024\227^1m&\233\273\2378;1\371\257k,\232i\177}u\304M\254\217\2522\316\030\354v\311\362\010\277\300t\345\206uX!\304" signing_key_seq_num: 0 } } request_id { client_id: "83b378faa4664cb59dbd1e93d46782e0" seq_no: 383 first_incomplete_seq_no: 383 attempt_no: 0 }
I20240502 01:53:35.912067 15982 raft_consensus.cc:2258] T 71b0f9fbee82464994252126b2d631bc P 8446b7a7ea67486288d174eea7a3b3c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:35.914656 15982 ts_tablet_manager.cc:1875] T 71b0f9fbee82464994252126b2d631bc P 8446b7a7ea67486288d174eea7a3b3c6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:53:35.924921 15982 ts_tablet_manager.cc:1888] T 71b0f9fbee82464994252126b2d631bc P 8446b7a7ea67486288d174eea7a3b3c6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.128
I20240502 01:53:35.925276 15982 log.cc:1196] T 71b0f9fbee82464994252126b2d631bc P 8446b7a7ea67486288d174eea7a3b3c6: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/wal/wals/71b0f9fbee82464994252126b2d631bc
I20240502 01:53:35.926648 13310 catalog_manager.cc:4830] TS 8446b7a7ea67486288d174eea7a3b3c6 (127.9.176.194:34851): tablet 71b0f9fbee82464994252126b2d631bc (table rebalance_test_table_4 [id=7c97435ee22d42aaa3452b6e2abdaf75]) successfully deleted
I20240502 01:53:36.545122 13451 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 01:53:36.581514 13539 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.176.198:38527, user_credentials={real_user=slave}} blocked reactor thread for 60199.4us
I20240502 01:53:36.619369 14934 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:36.640566 15208 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:36.680601 13715 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:53:36.692763 13584 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:53:36.735579 15067 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:36.772111 14799 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:36.962572 13470 consensus_queue.cc:237] T eb1cf83294e044b49e115323de67ed42 P 5359629f948641f4b56e35ad6b6a15c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 170, Committed index: 170, Last appended: 1.170, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 171 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } attrs { replace: true } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false } } peers { permanent_uuid: "7505fefeaf804ed8aeb3672f386ab959" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 38645 } attrs { promote: true } }
W20240502 01:53:36.965531 13405 consensus_peers.cc:485] T eb1cf83294e044b49e115323de67ed42 P 5359629f948641f4b56e35ad6b6a15c4 -> Peer 7505fefeaf804ed8aeb3672f386ab959 (127.9.176.199:38645): Couldn't send request to peer 7505fefeaf804ed8aeb3672f386ab959. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: eb1cf83294e044b49e115323de67ed42. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:36.965633 15084 raft_consensus.cc:1261] T eb1cf83294e044b49e115323de67ed42 P 9e4fa889fab047baa5bd3dcd77b2ca8f [term 1 FOLLOWER]: Refusing update from remote peer 5359629f948641f4b56e35ad6b6a15c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 170. Preceding OpId from leader: term: 1 index: 171. (index mismatch)
I20240502 01:53:36.966246 15643 consensus_queue.cc:1035] T eb1cf83294e044b49e115323de67ed42 P 5359629f948641f4b56e35ad6b6a15c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 171, Last known committed idx: 170, Time since last communication: 0.000s
I20240502 01:53:37.000989 15817 raft_consensus.cc:2932] T eb1cf83294e044b49e115323de67ed42 P 5359629f948641f4b56e35ad6b6a15c4 [term 1 LEADER]: Committing config change with OpId 1.171: config changed from index 77 to 171, NON_VOTER 7505fefeaf804ed8aeb3672f386ab959 (127.9.176.199) added. New config: { opid_index: 171 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } attrs { replace: true } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false } } peers { permanent_uuid: "7505fefeaf804ed8aeb3672f386ab959" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 38645 } attrs { promote: true } } }
I20240502 01:53:37.005869 15084 raft_consensus.cc:2932] T eb1cf83294e044b49e115323de67ed42 P 9e4fa889fab047baa5bd3dcd77b2ca8f [term 1 FOLLOWER]: Committing config change with OpId 1.171: config changed from index 77 to 171, NON_VOTER 7505fefeaf804ed8aeb3672f386ab959 (127.9.176.199) added. New config: { opid_index: 171 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } attrs { replace: true } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false } } peers { permanent_uuid: "7505fefeaf804ed8aeb3672f386ab959" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 38645 } attrs { promote: true } } }
I20240502 01:53:37.011477 13325 catalog_manager.cc:5484] T eb1cf83294e044b49e115323de67ed42 P 5359629f948641f4b56e35ad6b6a15c4 reported cstate change: config changed from index 77 to 171, NON_VOTER 7505fefeaf804ed8aeb3672f386ab959 (127.9.176.199) added. New cstate: current_term: 1 leader_uuid: "5359629f948641f4b56e35ad6b6a15c4" committed_config { opid_index: 171 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7505fefeaf804ed8aeb3672f386ab959" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 38645 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:37.086434 13598 consensus_queue.cc:237] T 930645182ab748d7b864fece1e858b3f P 8446b7a7ea67486288d174eea7a3b3c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 142, Committed index: 142, Last appended: 1.142, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 143 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false replace: true } } peers { permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 41329 } attrs { promote: true } }
I20240502 01:53:37.090984 15084 raft_consensus.cc:1261] T 930645182ab748d7b864fece1e858b3f P 9e4fa889fab047baa5bd3dcd77b2ca8f [term 1 FOLLOWER]: Refusing update from remote peer 8446b7a7ea67486288d174eea7a3b3c6: Log matching property violated. Preceding OpId in replica: term: 1 index: 142. Preceding OpId from leader: term: 1 index: 143. (index mismatch)
I20240502 01:53:37.091758 15849 consensus_queue.cc:1035] T 930645182ab748d7b864fece1e858b3f P 8446b7a7ea67486288d174eea7a3b3c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 143, Last known committed idx: 142, Time since last communication: 0.000s
I20240502 01:53:37.095085 15498 raft_consensus.cc:2932] T 930645182ab748d7b864fece1e858b3f P 8446b7a7ea67486288d174eea7a3b3c6 [term 1 LEADER]: Committing config change with OpId 1.143: config changed from index 89 to 143, NON_VOTER ce90a4bd06ba41dd8e5aa227de9780a5 (127.9.176.196) added. New config: { opid_index: 143 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false replace: true } } peers { permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 41329 } attrs { promote: true } } }
I20240502 01:53:37.095840 15086 raft_consensus.cc:2932] T 930645182ab748d7b864fece1e858b3f P 9e4fa889fab047baa5bd3dcd77b2ca8f [term 1 FOLLOWER]: Committing config change with OpId 1.143: config changed from index 89 to 143, NON_VOTER ce90a4bd06ba41dd8e5aa227de9780a5 (127.9.176.196) added. New config: { opid_index: 143 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false replace: true } } peers { permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 41329 } attrs { promote: true } } }
I20240502 01:53:37.103343 13318 catalog_manager.cc:5484] T 930645182ab748d7b864fece1e858b3f P 8446b7a7ea67486288d174eea7a3b3c6 reported cstate change: config changed from index 89 to 143, NON_VOTER ce90a4bd06ba41dd8e5aa227de9780a5 (127.9.176.196) added. New cstate: current_term: 1 leader_uuid: "8446b7a7ea67486288d174eea7a3b3c6" committed_config { opid_index: 143 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 41329 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 01:53:37.119508 13536 consensus_peers.cc:485] T 930645182ab748d7b864fece1e858b3f P 8446b7a7ea67486288d174eea7a3b3c6 -> Peer ce90a4bd06ba41dd8e5aa227de9780a5 (127.9.176.196:41329): Couldn't send request to peer ce90a4bd06ba41dd8e5aa227de9780a5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 930645182ab748d7b864fece1e858b3f. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:37.182335 13470 consensus_queue.cc:237] T b38a1fb4d1694c609a932e8816fc3514 P 5359629f948641f4b56e35ad6b6a15c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 156, Committed index: 156, Last appended: 1.156, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 157 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } attrs { replace: true } } peers { permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: true } }
I20240502 01:53:37.185086 14952 raft_consensus.cc:1261] T b38a1fb4d1694c609a932e8816fc3514 P 7e74a3d5df6a483c95c9dabfc99577bc [term 1 FOLLOWER]: Refusing update from remote peer 5359629f948641f4b56e35ad6b6a15c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 156. Preceding OpId from leader: term: 1 index: 157. (index mismatch)
W20240502 01:53:37.185799 13405 consensus_peers.cc:485] T b38a1fb4d1694c609a932e8816fc3514 P 5359629f948641f4b56e35ad6b6a15c4 -> Peer 9e4fa889fab047baa5bd3dcd77b2ca8f (127.9.176.198:38527): Couldn't send request to peer 9e4fa889fab047baa5bd3dcd77b2ca8f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b38a1fb4d1694c609a932e8816fc3514. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:37.186842 15356 consensus_queue.cc:1035] T b38a1fb4d1694c609a932e8816fc3514 P 5359629f948641f4b56e35ad6b6a15c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 157, Last known committed idx: 156, Time since last communication: 0.000s
I20240502 01:53:37.190984 15817 raft_consensus.cc:2932] T b38a1fb4d1694c609a932e8816fc3514 P 5359629f948641f4b56e35ad6b6a15c4 [term 1 LEADER]: Committing config change with OpId 1.157: config changed from index 84 to 157, NON_VOTER 9e4fa889fab047baa5bd3dcd77b2ca8f (127.9.176.198) added. New config: { opid_index: 157 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } attrs { replace: true } } peers { permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: true } } }
I20240502 01:53:37.194255 14952 raft_consensus.cc:2932] T b38a1fb4d1694c609a932e8816fc3514 P 7e74a3d5df6a483c95c9dabfc99577bc [term 1 FOLLOWER]: Committing config change with OpId 1.157: config changed from index 84 to 157, NON_VOTER 9e4fa889fab047baa5bd3dcd77b2ca8f (127.9.176.198) added. New config: { opid_index: 157 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } attrs { replace: true } } peers { permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: true } } }
I20240502 01:53:37.205264 13325 catalog_manager.cc:5484] T b38a1fb4d1694c609a932e8816fc3514 P 5359629f948641f4b56e35ad6b6a15c4 reported cstate change: config changed from index 84 to 157, NON_VOTER 9e4fa889fab047baa5bd3dcd77b2ca8f (127.9.176.198) added. New cstate: current_term: 1 leader_uuid: "5359629f948641f4b56e35ad6b6a15c4" committed_config { opid_index: 157 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:37.557595 16088 ts_tablet_manager.cc:902] T eb1cf83294e044b49e115323de67ed42 P 7505fefeaf804ed8aeb3672f386ab959: Initiating tablet copy from peer 5359629f948641f4b56e35ad6b6a15c4 (127.9.176.193:33849)
I20240502 01:53:37.591094 16092 ts_tablet_manager.cc:902] T b38a1fb4d1694c609a932e8816fc3514 P 9e4fa889fab047baa5bd3dcd77b2ca8f: Initiating tablet copy from peer 5359629f948641f4b56e35ad6b6a15c4 (127.9.176.193:33849)
I20240502 01:53:37.610860 16088 tablet_copy_client.cc:308] T eb1cf83294e044b49e115323de67ed42 P 7505fefeaf804ed8aeb3672f386ab959: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.193:33849
I20240502 01:53:37.611904 13491 tablet_copy_service.cc:140] P 5359629f948641f4b56e35ad6b6a15c4: Received BeginTabletCopySession request for tablet eb1cf83294e044b49e115323de67ed42 from peer 7505fefeaf804ed8aeb3672f386ab959 ({username='slave'} at 127.9.176.199:45881)
I20240502 01:53:37.612138 13491 tablet_copy_service.cc:161] P 5359629f948641f4b56e35ad6b6a15c4: Beginning new tablet copy session on tablet eb1cf83294e044b49e115323de67ed42 from peer 7505fefeaf804ed8aeb3672f386ab959 at {username='slave'} at 127.9.176.199:45881: session id = 7505fefeaf804ed8aeb3672f386ab959-eb1cf83294e044b49e115323de67ed42
I20240502 01:53:37.616163 13491 tablet_copy_source_session.cc:205] T eb1cf83294e044b49e115323de67ed42 P 5359629f948641f4b56e35ad6b6a15c4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:37.620932 16088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb1cf83294e044b49e115323de67ed42. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:37.627936 16088 tablet_copy_client.cc:783] T eb1cf83294e044b49e115323de67ed42 P 7505fefeaf804ed8aeb3672f386ab959: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:37.628535 16088 tablet_copy_client.cc:647] T eb1cf83294e044b49e115323de67ed42 P 7505fefeaf804ed8aeb3672f386ab959: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:37.632382 16088 tablet_copy_client.cc:523] T eb1cf83294e044b49e115323de67ed42 P 7505fefeaf804ed8aeb3672f386ab959: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:37.635190 16092 tablet_copy_client.cc:308] T b38a1fb4d1694c609a932e8816fc3514 P 9e4fa889fab047baa5bd3dcd77b2ca8f: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.193:33849
I20240502 01:53:37.636464 13491 tablet_copy_service.cc:140] P 5359629f948641f4b56e35ad6b6a15c4: Received BeginTabletCopySession request for tablet b38a1fb4d1694c609a932e8816fc3514 from peer 9e4fa889fab047baa5bd3dcd77b2ca8f ({username='slave'} at 127.9.176.198:54295)
I20240502 01:53:37.636670 13491 tablet_copy_service.cc:161] P 5359629f948641f4b56e35ad6b6a15c4: Beginning new tablet copy session on tablet b38a1fb4d1694c609a932e8816fc3514 from peer 9e4fa889fab047baa5bd3dcd77b2ca8f at {username='slave'} at 127.9.176.198:54295: session id = 9e4fa889fab047baa5bd3dcd77b2ca8f-b38a1fb4d1694c609a932e8816fc3514
I20240502 01:53:37.638152 16088 tablet_bootstrap.cc:492] T eb1cf83294e044b49e115323de67ed42 P 7505fefeaf804ed8aeb3672f386ab959: Bootstrap starting.
I20240502 01:53:37.639562 13491 tablet_copy_source_session.cc:205] T b38a1fb4d1694c609a932e8816fc3514 P 5359629f948641f4b56e35ad6b6a15c4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:37.648164 16092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b38a1fb4d1694c609a932e8816fc3514. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:37.688025 16099 ts_tablet_manager.cc:902] T 930645182ab748d7b864fece1e858b3f P ce90a4bd06ba41dd8e5aa227de9780a5: Initiating tablet copy from peer 8446b7a7ea67486288d174eea7a3b3c6 (127.9.176.194:34851)
I20240502 01:53:37.705547 16092 tablet_copy_client.cc:783] T b38a1fb4d1694c609a932e8816fc3514 P 9e4fa889fab047baa5bd3dcd77b2ca8f: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:37.705888 16092 tablet_copy_client.cc:647] T b38a1fb4d1694c609a932e8816fc3514 P 9e4fa889fab047baa5bd3dcd77b2ca8f: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:37.709089 16092 tablet_copy_client.cc:523] T b38a1fb4d1694c609a932e8816fc3514 P 9e4fa889fab047baa5bd3dcd77b2ca8f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:37.713177 16092 tablet_bootstrap.cc:492] T b38a1fb4d1694c609a932e8816fc3514 P 9e4fa889fab047baa5bd3dcd77b2ca8f: Bootstrap starting.
W20240502 01:53:37.782333 13402 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.176.196:41329, user_credentials={real_user=slave}} blocked reactor thread for 77002.3us
I20240502 01:53:37.800125 16099 tablet_copy_client.cc:308] T 930645182ab748d7b864fece1e858b3f P ce90a4bd06ba41dd8e5aa227de9780a5: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.194:34851
I20240502 01:53:37.825984 13624 tablet_copy_service.cc:140] P 8446b7a7ea67486288d174eea7a3b3c6: Received BeginTabletCopySession request for tablet 930645182ab748d7b864fece1e858b3f from peer ce90a4bd06ba41dd8e5aa227de9780a5 ({username='slave'} at 127.9.176.196:53687)
I20240502 01:53:37.826185 13624 tablet_copy_service.cc:161] P 8446b7a7ea67486288d174eea7a3b3c6: Beginning new tablet copy session on tablet 930645182ab748d7b864fece1e858b3f from peer ce90a4bd06ba41dd8e5aa227de9780a5 at {username='slave'} at 127.9.176.196:53687: session id = ce90a4bd06ba41dd8e5aa227de9780a5-930645182ab748d7b864fece1e858b3f
I20240502 01:53:37.828863 13624 tablet_copy_source_session.cc:205] T 930645182ab748d7b864fece1e858b3f P 8446b7a7ea67486288d174eea7a3b3c6: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:37.844798 16099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 930645182ab748d7b864fece1e858b3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:37.854593 16099 tablet_copy_client.cc:783] T 930645182ab748d7b864fece1e858b3f P ce90a4bd06ba41dd8e5aa227de9780a5: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:37.855021 16099 tablet_copy_client.cc:647] T 930645182ab748d7b864fece1e858b3f P ce90a4bd06ba41dd8e5aa227de9780a5: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:37.862567 16099 tablet_copy_client.cc:523] T 930645182ab748d7b864fece1e858b3f P ce90a4bd06ba41dd8e5aa227de9780a5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:37.914645 16099 tablet_bootstrap.cc:492] T 930645182ab748d7b864fece1e858b3f P ce90a4bd06ba41dd8e5aa227de9780a5: Bootstrap starting.
I20240502 01:53:38.277297 16092 tablet_bootstrap.cc:492] T b38a1fb4d1694c609a932e8816fc3514 P 9e4fa889fab047baa5bd3dcd77b2ca8f: Bootstrap replayed 1/1 log segments. Stats: ops{read=162 overwritten=0 applied=161 ignored=0} inserts{seen=789 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 01:53:38.277848 16092 tablet_bootstrap.cc:492] T b38a1fb4d1694c609a932e8816fc3514 P 9e4fa889fab047baa5bd3dcd77b2ca8f: Bootstrap complete.
I20240502 01:53:38.278193 16092 ts_tablet_manager.cc:1367] T b38a1fb4d1694c609a932e8816fc3514 P 9e4fa889fab047baa5bd3dcd77b2ca8f: Time spent bootstrapping tablet: real 0.565s	user 0.237s	sys 0.025s
I20240502 01:53:38.279022 16092 raft_consensus.cc:348] T b38a1fb4d1694c609a932e8816fc3514 P 9e4fa889fab047baa5bd3dcd77b2ca8f [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 157 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } attrs { replace: true } } peers { permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: true } }
I20240502 01:53:38.280040 16092 raft_consensus.cc:729] T b38a1fb4d1694c609a932e8816fc3514 P 9e4fa889fab047baa5bd3dcd77b2ca8f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9e4fa889fab047baa5bd3dcd77b2ca8f, State: Initialized, Role: LEARNER
I20240502 01:53:38.280313 16092 consensus_queue.cc:260] T b38a1fb4d1694c609a932e8816fc3514 P 9e4fa889fab047baa5bd3dcd77b2ca8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 161, Last appended: 1.162, Last appended by leader: 162, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 157 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } attrs { replace: true } } peers { permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: true } }
I20240502 01:53:38.287981 16092 ts_tablet_manager.cc:1398] T b38a1fb4d1694c609a932e8816fc3514 P 9e4fa889fab047baa5bd3dcd77b2ca8f: Time spent starting tablet: real 0.010s	user 0.006s	sys 0.004s
I20240502 01:53:38.296351 13491 tablet_copy_service.cc:342] P 5359629f948641f4b56e35ad6b6a15c4: Request end of tablet copy session 9e4fa889fab047baa5bd3dcd77b2ca8f-b38a1fb4d1694c609a932e8816fc3514 received from {username='slave'} at 127.9.176.198:54295
I20240502 01:53:38.296540 13491 tablet_copy_service.cc:433] P 5359629f948641f4b56e35ad6b6a15c4: ending tablet copy session 9e4fa889fab047baa5bd3dcd77b2ca8f-b38a1fb4d1694c609a932e8816fc3514 on tablet b38a1fb4d1694c609a932e8816fc3514 with peer 9e4fa889fab047baa5bd3dcd77b2ca8f
I20240502 01:53:38.313647 16088 tablet_bootstrap.cc:492] T eb1cf83294e044b49e115323de67ed42 P 7505fefeaf804ed8aeb3672f386ab959: Bootstrap replayed 1/1 log segments. Stats: ops{read=181 overwritten=0 applied=181 ignored=0} inserts{seen=863 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:53:38.314103 16088 tablet_bootstrap.cc:492] T eb1cf83294e044b49e115323de67ed42 P 7505fefeaf804ed8aeb3672f386ab959: Bootstrap complete.
I20240502 01:53:38.314381 16088 ts_tablet_manager.cc:1367] T eb1cf83294e044b49e115323de67ed42 P 7505fefeaf804ed8aeb3672f386ab959: Time spent bootstrapping tablet: real 0.676s	user 0.256s	sys 0.016s
I20240502 01:53:38.315212 16088 raft_consensus.cc:348] T eb1cf83294e044b49e115323de67ed42 P 7505fefeaf804ed8aeb3672f386ab959 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 171 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } attrs { replace: true } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false } } peers { permanent_uuid: "7505fefeaf804ed8aeb3672f386ab959" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 38645 } attrs { promote: true } }
I20240502 01:53:38.315491 16088 raft_consensus.cc:729] T eb1cf83294e044b49e115323de67ed42 P 7505fefeaf804ed8aeb3672f386ab959 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7505fefeaf804ed8aeb3672f386ab959, State: Initialized, Role: LEARNER
I20240502 01:53:38.315831 16088 consensus_queue.cc:260] T eb1cf83294e044b49e115323de67ed42 P 7505fefeaf804ed8aeb3672f386ab959 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 181, Last appended: 1.181, Last appended by leader: 181, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 171 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } attrs { replace: true } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false } } peers { permanent_uuid: "7505fefeaf804ed8aeb3672f386ab959" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 38645 } attrs { promote: true } }
I20240502 01:53:38.316630 16088 ts_tablet_manager.cc:1398] T eb1cf83294e044b49e115323de67ed42 P 7505fefeaf804ed8aeb3672f386ab959: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 01:53:38.317548 13491 tablet_copy_service.cc:342] P 5359629f948641f4b56e35ad6b6a15c4: Request end of tablet copy session 7505fefeaf804ed8aeb3672f386ab959-eb1cf83294e044b49e115323de67ed42 received from {username='slave'} at 127.9.176.199:45881
I20240502 01:53:38.317719 13491 tablet_copy_service.cc:433] P 5359629f948641f4b56e35ad6b6a15c4: ending tablet copy session 7505fefeaf804ed8aeb3672f386ab959-eb1cf83294e044b49e115323de67ed42 on tablet eb1cf83294e044b49e115323de67ed42 with peer 7505fefeaf804ed8aeb3672f386ab959
I20240502 01:53:38.668939 15225 raft_consensus.cc:1203] T eb1cf83294e044b49e115323de67ed42 P 7505fefeaf804ed8aeb3672f386ab959 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.170->[1.171-1.196]   Dedup: 1.181->[1.182-1.196]
I20240502 01:53:38.688630 15086 raft_consensus.cc:1203] T b38a1fb4d1694c609a932e8816fc3514 P 9e4fa889fab047baa5bd3dcd77b2ca8f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.156->[1.157-1.173]   Dedup: 1.162->[1.163-1.173]
I20240502 01:53:38.786091 15817 raft_consensus.cc:1050] T b38a1fb4d1694c609a932e8816fc3514 P 5359629f948641f4b56e35ad6b6a15c4: attempting to promote NON_VOTER 9e4fa889fab047baa5bd3dcd77b2ca8f to VOTER
I20240502 01:53:38.786880 15817 consensus_queue.cc:237] T b38a1fb4d1694c609a932e8816fc3514 P 5359629f948641f4b56e35ad6b6a15c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 176, Committed index: 176, Last appended: 1.176, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 177 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } attrs { replace: true } } peers { permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false } }
I20240502 01:53:38.789896 14952 raft_consensus.cc:1261] T b38a1fb4d1694c609a932e8816fc3514 P 7e74a3d5df6a483c95c9dabfc99577bc [term 1 FOLLOWER]: Refusing update from remote peer 5359629f948641f4b56e35ad6b6a15c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 176. Preceding OpId from leader: term: 1 index: 177. (index mismatch)
I20240502 01:53:38.791467 15643 consensus_queue.cc:1035] T b38a1fb4d1694c609a932e8816fc3514 P 5359629f948641f4b56e35ad6b6a15c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 177, Last known committed idx: 176, Time since last communication: 0.000s
I20240502 01:53:38.798192 15356 raft_consensus.cc:2932] T b38a1fb4d1694c609a932e8816fc3514 P 5359629f948641f4b56e35ad6b6a15c4 [term 1 LEADER]: Committing config change with OpId 1.177: config changed from index 157 to 177, 9e4fa889fab047baa5bd3dcd77b2ca8f (127.9.176.198) changed from NON_VOTER to VOTER. New config: { opid_index: 177 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } attrs { replace: true } } peers { permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false } } }
I20240502 01:53:38.805289 13325 catalog_manager.cc:5484] T b38a1fb4d1694c609a932e8816fc3514 P 5359629f948641f4b56e35ad6b6a15c4 reported cstate change: config changed from index 157 to 177, 9e4fa889fab047baa5bd3dcd77b2ca8f (127.9.176.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5359629f948641f4b56e35ad6b6a15c4" committed_config { opid_index: 177 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:38.807315 14954 raft_consensus.cc:2932] T b38a1fb4d1694c609a932e8816fc3514 P 7e74a3d5df6a483c95c9dabfc99577bc [term 1 FOLLOWER]: Committing config change with OpId 1.177: config changed from index 157 to 177, 9e4fa889fab047baa5bd3dcd77b2ca8f (127.9.176.198) changed from NON_VOTER to VOTER. New config: { opid_index: 177 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } attrs { replace: true } } peers { permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false } } }
I20240502 01:53:38.815960 16099 tablet_bootstrap.cc:492] T 930645182ab748d7b864fece1e858b3f P ce90a4bd06ba41dd8e5aa227de9780a5: Bootstrap replayed 1/1 log segments. Stats: ops{read=155 overwritten=0 applied=155 ignored=0} inserts{seen=837 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:53:38.818483 16099 tablet_bootstrap.cc:492] T 930645182ab748d7b864fece1e858b3f P ce90a4bd06ba41dd8e5aa227de9780a5: Bootstrap complete.
I20240502 01:53:38.819479 16099 ts_tablet_manager.cc:1367] T 930645182ab748d7b864fece1e858b3f P ce90a4bd06ba41dd8e5aa227de9780a5: Time spent bootstrapping tablet: real 0.938s	user 0.214s	sys 0.024s
I20240502 01:53:38.821178 16099 raft_consensus.cc:348] T 930645182ab748d7b864fece1e858b3f P ce90a4bd06ba41dd8e5aa227de9780a5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 143 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false replace: true } } peers { permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 41329 } attrs { promote: true } }
I20240502 01:53:38.824671 16099 raft_consensus.cc:729] T 930645182ab748d7b864fece1e858b3f P ce90a4bd06ba41dd8e5aa227de9780a5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ce90a4bd06ba41dd8e5aa227de9780a5, State: Initialized, Role: LEARNER
I20240502 01:53:38.825065 16099 consensus_queue.cc:260] T 930645182ab748d7b864fece1e858b3f P ce90a4bd06ba41dd8e5aa227de9780a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 155, Last appended: 1.155, Last appended by leader: 155, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 143 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false replace: true } } peers { permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 41329 } attrs { promote: true } }
I20240502 01:53:38.829429 16099 ts_tablet_manager.cc:1398] T 930645182ab748d7b864fece1e858b3f P ce90a4bd06ba41dd8e5aa227de9780a5: Time spent starting tablet: real 0.009s	user 0.000s	sys 0.002s
I20240502 01:53:38.833413 13624 tablet_copy_service.cc:342] P 8446b7a7ea67486288d174eea7a3b3c6: Request end of tablet copy session ce90a4bd06ba41dd8e5aa227de9780a5-930645182ab748d7b864fece1e858b3f received from {username='slave'} at 127.9.176.196:53687
I20240502 01:53:38.833607 13624 tablet_copy_service.cc:433] P 8446b7a7ea67486288d174eea7a3b3c6: ending tablet copy session ce90a4bd06ba41dd8e5aa227de9780a5-930645182ab748d7b864fece1e858b3f on tablet 930645182ab748d7b864fece1e858b3f with peer ce90a4bd06ba41dd8e5aa227de9780a5
I20240502 01:53:38.835608 15996 raft_consensus.cc:1050] T eb1cf83294e044b49e115323de67ed42 P 5359629f948641f4b56e35ad6b6a15c4: attempting to promote NON_VOTER 7505fefeaf804ed8aeb3672f386ab959 to VOTER
I20240502 01:53:38.836421 15996 consensus_queue.cc:237] T eb1cf83294e044b49e115323de67ed42 P 5359629f948641f4b56e35ad6b6a15c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 198, Committed index: 198, Last appended: 1.199, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 200 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } attrs { replace: true } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false } } peers { permanent_uuid: "7505fefeaf804ed8aeb3672f386ab959" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 38645 } attrs { promote: false } }
I20240502 01:53:38.841820 15084 raft_consensus.cc:1261] T eb1cf83294e044b49e115323de67ed42 P 9e4fa889fab047baa5bd3dcd77b2ca8f [term 1 FOLLOWER]: Refusing update from remote peer 5359629f948641f4b56e35ad6b6a15c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 198. Preceding OpId from leader: term: 1 index: 200. (index mismatch)
I20240502 01:53:38.843335 15086 raft_consensus.cc:1261] T b38a1fb4d1694c609a932e8816fc3514 P 9e4fa889fab047baa5bd3dcd77b2ca8f [term 1 LEARNER]: Refusing update from remote peer 5359629f948641f4b56e35ad6b6a15c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 176. Preceding OpId from leader: term: 1 index: 177. (index mismatch)
I20240502 01:53:38.844199 15225 raft_consensus.cc:1261] T eb1cf83294e044b49e115323de67ed42 P 7505fefeaf804ed8aeb3672f386ab959 [term 1 LEARNER]: Refusing update from remote peer 5359629f948641f4b56e35ad6b6a15c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 199. Preceding OpId from leader: term: 1 index: 200. (index mismatch)
I20240502 01:53:38.845697 15356 consensus_queue.cc:1035] T eb1cf83294e044b49e115323de67ed42 P 5359629f948641f4b56e35ad6b6a15c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7505fefeaf804ed8aeb3672f386ab959" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 38645 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 200, Last known committed idx: 198, Time since last communication: 0.000s
I20240502 01:53:38.848784 15643 raft_consensus.cc:2932] T eb1cf83294e044b49e115323de67ed42 P 5359629f948641f4b56e35ad6b6a15c4 [term 1 LEADER]: Committing config change with OpId 1.200: config changed from index 171 to 200, 7505fefeaf804ed8aeb3672f386ab959 (127.9.176.199) changed from NON_VOTER to VOTER. New config: { opid_index: 200 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } attrs { replace: true } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false } } peers { permanent_uuid: "7505fefeaf804ed8aeb3672f386ab959" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 38645 } attrs { promote: false } } }
I20240502 01:53:38.851267 15996 consensus_queue.cc:1035] T eb1cf83294e044b49e115323de67ed42 P 5359629f948641f4b56e35ad6b6a15c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 200, Last known committed idx: 198, Time since last communication: 0.000s
I20240502 01:53:38.851385 15817 consensus_queue.cc:1035] T b38a1fb4d1694c609a932e8816fc3514 P 5359629f948641f4b56e35ad6b6a15c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 177, Last known committed idx: 176, Time since last communication: 0.000s
I20240502 01:53:38.854413 13318 catalog_manager.cc:5484] T eb1cf83294e044b49e115323de67ed42 P 5359629f948641f4b56e35ad6b6a15c4 reported cstate change: config changed from index 171 to 200, 7505fefeaf804ed8aeb3672f386ab959 (127.9.176.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5359629f948641f4b56e35ad6b6a15c4" committed_config { opid_index: 200 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7505fefeaf804ed8aeb3672f386ab959" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 38645 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:38.855250 15226 raft_consensus.cc:2932] T eb1cf83294e044b49e115323de67ed42 P 7505fefeaf804ed8aeb3672f386ab959 [term 1 FOLLOWER]: Committing config change with OpId 1.200: config changed from index 171 to 200, 7505fefeaf804ed8aeb3672f386ab959 (127.9.176.199) changed from NON_VOTER to VOTER. New config: { opid_index: 200 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } attrs { replace: true } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false } } peers { permanent_uuid: "7505fefeaf804ed8aeb3672f386ab959" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 38645 } attrs { promote: false } } }
I20240502 01:53:38.872849 15085 raft_consensus.cc:2932] T eb1cf83294e044b49e115323de67ed42 P 9e4fa889fab047baa5bd3dcd77b2ca8f [term 1 FOLLOWER]: Committing config change with OpId 1.200: config changed from index 171 to 200, 7505fefeaf804ed8aeb3672f386ab959 (127.9.176.199) changed from NON_VOTER to VOTER. New config: { opid_index: 200 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } attrs { replace: true } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false } } peers { permanent_uuid: "7505fefeaf804ed8aeb3672f386ab959" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 38645 } attrs { promote: false } } }
I20240502 01:53:38.887750 15085 raft_consensus.cc:2932] T b38a1fb4d1694c609a932e8816fc3514 P 9e4fa889fab047baa5bd3dcd77b2ca8f [term 1 FOLLOWER]: Committing config change with OpId 1.177: config changed from index 157 to 177, 9e4fa889fab047baa5bd3dcd77b2ca8f (127.9.176.198) changed from NON_VOTER to VOTER. New config: { opid_index: 177 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } attrs { replace: true } } peers { permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false } } }
I20240502 01:53:39.078337 13470 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "eb1cf83294e044b49e115323de67ed42"
dest_uuid: "5359629f948641f4b56e35ad6b6a15c4"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:38266
I20240502 01:53:39.078655 13470 raft_consensus.cc:595] T eb1cf83294e044b49e115323de67ed42 P 5359629f948641f4b56e35ad6b6a15c4 [term 1 LEADER]: Received request to transfer leadership
W20240502 01:53:39.102002 13426 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:36168: Service unavailable: leader transfer in progress
W20240502 01:53:39.106236 13426 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:36168: Service unavailable: leader transfer in progress
W20240502 01:53:39.115296 13430 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:36168: Service unavailable: leader transfer in progress
W20240502 01:53:39.124032 13430 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:36168: Service unavailable: leader transfer in progress
W20240502 01:53:39.134972 13430 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:36168: Service unavailable: leader transfer in progress
W20240502 01:53:39.147979 13430 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:36168: Service unavailable: leader transfer in progress
W20240502 01:53:39.158926 13430 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:36168: Service unavailable: leader transfer in progress
W20240502 01:53:39.176921 13427 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:36168: Service unavailable: leader transfer in progress
W20240502 01:53:39.190358 13427 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:36168: Service unavailable: leader transfer in progress
W20240502 01:53:39.206538 13427 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:36168: Service unavailable: leader transfer in progress
W20240502 01:53:39.225699 13427 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:36168: Service unavailable: leader transfer in progress
W20240502 01:53:39.239686 13427 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:36168: Service unavailable: leader transfer in progress
W20240502 01:53:39.257329 13427 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:36168: Service unavailable: leader transfer in progress
W20240502 01:53:39.274540 13430 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:36168: Service unavailable: leader transfer in progress
W20240502 01:53:39.299060 13426 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:36168: Service unavailable: leader transfer in progress
I20240502 01:53:39.317127 14820 raft_consensus.cc:1203] T 930645182ab748d7b864fece1e858b3f P ce90a4bd06ba41dd8e5aa227de9780a5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.142->[1.143-1.178]   Dedup: 1.155->[1.156-1.178]
W20240502 01:53:39.320219 13426 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:36168: Service unavailable: leader transfer in progress
W20240502 01:53:39.339342 13427 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:36168: Service unavailable: leader transfer in progress
W20240502 01:53:39.363209 13427 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:36168: Service unavailable: leader transfer in progress
I20240502 01:53:39.387238 16065 raft_consensus.cc:1050] T 930645182ab748d7b864fece1e858b3f P 8446b7a7ea67486288d174eea7a3b3c6: attempting to promote NON_VOTER ce90a4bd06ba41dd8e5aa227de9780a5 to VOTER
W20240502 01:53:39.389433 13426 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:36168: Service unavailable: leader transfer in progress
I20240502 01:53:39.391252 16065 consensus_queue.cc:237] T 930645182ab748d7b864fece1e858b3f P 8446b7a7ea67486288d174eea7a3b3c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 180, Committed index: 180, Last appended: 1.181, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 182 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false replace: true } } peers { permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 41329 } attrs { promote: false } }
W20240502 01:53:39.416184 13426 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:36168: Service unavailable: leader transfer in progress
I20240502 01:53:39.418213 14821 raft_consensus.cc:1261] T 930645182ab748d7b864fece1e858b3f P ce90a4bd06ba41dd8e5aa227de9780a5 [term 1 LEARNER]: Refusing update from remote peer 8446b7a7ea67486288d174eea7a3b3c6: Log matching property violated. Preceding OpId in replica: term: 1 index: 181. Preceding OpId from leader: term: 1 index: 182. (index mismatch)
I20240502 01:53:39.420985 16179 consensus_queue.cc:1035] T 930645182ab748d7b864fece1e858b3f P 8446b7a7ea67486288d174eea7a3b3c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 41329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 182, Last known committed idx: 180, Time since last communication: 0.000s
I20240502 01:53:39.429919 16181 raft_consensus.cc:2932] T 930645182ab748d7b864fece1e858b3f P 8446b7a7ea67486288d174eea7a3b3c6 [term 1 LEADER]: Committing config change with OpId 1.182: config changed from index 143 to 182, ce90a4bd06ba41dd8e5aa227de9780a5 (127.9.176.196) changed from NON_VOTER to VOTER. New config: { opid_index: 182 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false replace: true } } peers { permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 41329 } attrs { promote: false } } }
I20240502 01:53:39.430575 14821 raft_consensus.cc:2932] T 930645182ab748d7b864fece1e858b3f P ce90a4bd06ba41dd8e5aa227de9780a5 [term 1 FOLLOWER]: Committing config change with OpId 1.182: config changed from index 143 to 182, ce90a4bd06ba41dd8e5aa227de9780a5 (127.9.176.196) changed from NON_VOTER to VOTER. New config: { opid_index: 182 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false replace: true } } peers { permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 41329 } attrs { promote: false } } }
I20240502 01:53:39.431341 15084 raft_consensus.cc:1261] T 930645182ab748d7b864fece1e858b3f P 9e4fa889fab047baa5bd3dcd77b2ca8f [term 1 FOLLOWER]: Refusing update from remote peer 8446b7a7ea67486288d174eea7a3b3c6: Log matching property violated. Preceding OpId in replica: term: 1 index: 181. Preceding OpId from leader: term: 1 index: 182. (index mismatch)
I20240502 01:53:39.436400 13325 catalog_manager.cc:5484] T 930645182ab748d7b864fece1e858b3f P ce90a4bd06ba41dd8e5aa227de9780a5 reported cstate change: config changed from index 143 to 182, ce90a4bd06ba41dd8e5aa227de9780a5 (127.9.176.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8446b7a7ea67486288d174eea7a3b3c6" committed_config { opid_index: 182 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false replace: true } } peers { permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 41329 } attrs { promote: false } } }
I20240502 01:53:39.441293 16179 consensus_queue.cc:1035] T 930645182ab748d7b864fece1e858b3f P 8446b7a7ea67486288d174eea7a3b3c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 182, Last known committed idx: 181, Time since last communication: 0.000s
W20240502 01:53:39.441874 13426 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:36168: Service unavailable: leader transfer in progress
I20240502 01:53:39.446384 15085 raft_consensus.cc:2932] T 930645182ab748d7b864fece1e858b3f P 9e4fa889fab047baa5bd3dcd77b2ca8f [term 1 FOLLOWER]: Committing config change with OpId 1.182: config changed from index 143 to 182, ce90a4bd06ba41dd8e5aa227de9780a5 (127.9.176.196) changed from NON_VOTER to VOTER. New config: { opid_index: 182 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false replace: true } } peers { permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 41329 } attrs { promote: false } } }
W20240502 01:53:39.471365 13427 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:36168: Service unavailable: leader transfer in progress
I20240502 01:53:39.474604 15356 raft_consensus.cc:979] T eb1cf83294e044b49e115323de67ed42 P 5359629f948641f4b56e35ad6b6a15c4: : Instructing follower 9e4fa889fab047baa5bd3dcd77b2ca8f to start an election
I20240502 01:53:39.474844 15356 raft_consensus.cc:1067] T eb1cf83294e044b49e115323de67ed42 P 5359629f948641f4b56e35ad6b6a15c4 [term 1 LEADER]: Signalling peer 9e4fa889fab047baa5bd3dcd77b2ca8f to start an election
I20240502 01:53:39.476459 15084 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "eb1cf83294e044b49e115323de67ed42"
dest_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f"
 from {username='slave'} at 127.9.176.193:45431
I20240502 01:53:39.477000 15084 raft_consensus.cc:482] T eb1cf83294e044b49e115323de67ed42 P 9e4fa889fab047baa5bd3dcd77b2ca8f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:53:39.477339 15084 raft_consensus.cc:3037] T eb1cf83294e044b49e115323de67ed42 P 9e4fa889fab047baa5bd3dcd77b2ca8f [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:53:39.491449 15084 raft_consensus.cc:504] T eb1cf83294e044b49e115323de67ed42 P 9e4fa889fab047baa5bd3dcd77b2ca8f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 200 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } attrs { replace: true } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false } } peers { permanent_uuid: "7505fefeaf804ed8aeb3672f386ab959" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 38645 } attrs { promote: false } }
W20240502 01:53:39.497252 13426 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:36168: Service unavailable: leader transfer in progress
I20240502 01:53:39.501230 15084 leader_election.cc:288] T eb1cf83294e044b49e115323de67ed42 P 9e4fa889fab047baa5bd3dcd77b2ca8f [CANDIDATE]: Term 2 election: Requested vote from peers 5359629f948641f4b56e35ad6b6a15c4 (127.9.176.193:33849), 7505fefeaf804ed8aeb3672f386ab959 (127.9.176.199:38645)
I20240502 01:53:39.513001 13470 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b38a1fb4d1694c609a932e8816fc3514"
dest_uuid: "5359629f948641f4b56e35ad6b6a15c4"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:38290
I20240502 01:53:39.513309 13470 raft_consensus.cc:595] T b38a1fb4d1694c609a932e8816fc3514 P 5359629f948641f4b56e35ad6b6a15c4 [term 1 LEADER]: Received request to transfer leadership
I20240502 01:53:39.532224 13470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb1cf83294e044b49e115323de67ed42" candidate_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" candidate_term: 2 candidate_status { last_received { term: 1 index: 206 } } ignore_live_leader: true dest_uuid: "5359629f948641f4b56e35ad6b6a15c4"
I20240502 01:53:39.532532 13470 raft_consensus.cc:3032] T eb1cf83294e044b49e115323de67ed42 P 5359629f948641f4b56e35ad6b6a15c4 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:53:39.532702 13470 raft_consensus.cc:729] T eb1cf83294e044b49e115323de67ed42 P 5359629f948641f4b56e35ad6b6a15c4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5359629f948641f4b56e35ad6b6a15c4, State: Running, Role: LEADER
I20240502 01:53:39.533142 13470 consensus_queue.cc:260] T eb1cf83294e044b49e115323de67ed42 P 5359629f948641f4b56e35ad6b6a15c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 206, Committed index: 206, Last appended: 1.206, Last appended by leader: 206, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 200 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } attrs { replace: true } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false } } peers { permanent_uuid: "7505fefeaf804ed8aeb3672f386ab959" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 38645 } attrs { promote: false } }
I20240502 01:53:39.533738 13470 raft_consensus.cc:3037] T eb1cf83294e044b49e115323de67ed42 P 5359629f948641f4b56e35ad6b6a15c4 [term 1 FOLLOWER]: Advancing to term 2
W20240502 01:53:39.573599 13427 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:36168: Illegal state: replica 5359629f948641f4b56e35ad6b6a15c4 is not leader of this config: current role FOLLOWER
I20240502 01:53:39.576490 13470 raft_consensus.cc:2454] T eb1cf83294e044b49e115323de67ed42 P 5359629f948641f4b56e35ad6b6a15c4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e4fa889fab047baa5bd3dcd77b2ca8f in term 2.
I20240502 01:53:39.577618 13604 consensus_queue.cc:237] T 930645182ab748d7b864fece1e858b3f P 8446b7a7ea67486288d174eea7a3b3c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 185, Committed index: 185, Last appended: 1.185, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 186 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } } peers { permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 41329 } attrs { promote: false } }
I20240502 01:53:39.580781 14819 raft_consensus.cc:1261] T 930645182ab748d7b864fece1e858b3f P ce90a4bd06ba41dd8e5aa227de9780a5 [term 1 FOLLOWER]: Refusing update from remote peer 8446b7a7ea67486288d174eea7a3b3c6: Log matching property violated. Preceding OpId in replica: term: 1 index: 185. Preceding OpId from leader: term: 1 index: 186. (index mismatch)
I20240502 01:53:39.583170 16179 consensus_queue.cc:1035] T 930645182ab748d7b864fece1e858b3f P 8446b7a7ea67486288d174eea7a3b3c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 41329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 186, Last known committed idx: 185, Time since last communication: 0.002s
I20240502 01:53:39.589460 16180 raft_consensus.cc:2932] T 930645182ab748d7b864fece1e858b3f P 8446b7a7ea67486288d174eea7a3b3c6 [term 1 LEADER]: Committing config change with OpId 1.186: config changed from index 182 to 186, VOTER 9e4fa889fab047baa5bd3dcd77b2ca8f (127.9.176.198) evicted. New config: { opid_index: 186 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } } peers { permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 41329 } attrs { promote: false } } }
W20240502 01:53:39.590157 13426 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:36164: Service unavailable: leader transfer in progress
I20240502 01:53:39.590955 14819 raft_consensus.cc:2932] T 930645182ab748d7b864fece1e858b3f P ce90a4bd06ba41dd8e5aa227de9780a5 [term 1 FOLLOWER]: Committing config change with OpId 1.186: config changed from index 182 to 186, VOTER 9e4fa889fab047baa5bd3dcd77b2ca8f (127.9.176.198) evicted. New config: { opid_index: 186 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } } peers { permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 41329 } attrs { promote: false } } }
I20240502 01:53:39.593991 13310 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 930645182ab748d7b864fece1e858b3f with cas_config_opid_index 182: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:53:39.597898 13318 catalog_manager.cc:5484] T 930645182ab748d7b864fece1e858b3f P ce90a4bd06ba41dd8e5aa227de9780a5 reported cstate change: config changed from index 182 to 186, VOTER 9e4fa889fab047baa5bd3dcd77b2ca8f (127.9.176.198) evicted. New cstate: current_term: 1 leader_uuid: "8446b7a7ea67486288d174eea7a3b3c6" committed_config { opid_index: 186 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } } peers { permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 41329 } attrs { promote: false } } }
W20240502 01:53:39.630561 13426 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:36164: Service unavailable: leader transfer in progress
I20240502 01:53:39.631357 15022 leader_election.cc:302] T eb1cf83294e044b49e115323de67ed42 P 9e4fa889fab047baa5bd3dcd77b2ca8f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5359629f948641f4b56e35ad6b6a15c4, 9e4fa889fab047baa5bd3dcd77b2ca8f; no voters: 
I20240502 01:53:39.631863 16189 raft_consensus.cc:2781] T eb1cf83294e044b49e115323de67ed42 P 9e4fa889fab047baa5bd3dcd77b2ca8f [term 2 FOLLOWER]: Leader election won for term 2
W20240502 01:53:39.639955 13426 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:36164: Service unavailable: leader transfer in progress
I20240502 01:53:39.645623 16189 raft_consensus.cc:686] T eb1cf83294e044b49e115323de67ed42 P 9e4fa889fab047baa5bd3dcd77b2ca8f [term 2 LEADER]: Becoming Leader. State: Replica: 9e4fa889fab047baa5bd3dcd77b2ca8f, State: Running, Role: LEADER
I20240502 01:53:39.646718 15067 tablet_service.cc:1514] Processing DeleteTablet for tablet 930645182ab748d7b864fece1e858b3f with delete_type TABLET_DATA_TOMBSTONED (TS 9e4fa889fab047baa5bd3dcd77b2ca8f not found in new config with opid_index 186) from {username='slave'} at 127.0.0.1:55864
W20240502 01:53:39.647053 13426 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:36164: Service unavailable: leader transfer in progress
I20240502 01:53:39.646349 16189 consensus_queue.cc:237] T eb1cf83294e044b49e115323de67ed42 P 9e4fa889fab047baa5bd3dcd77b2ca8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 206, Committed index: 206, Last appended: 1.206, Last appended by leader: 206, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 200 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } attrs { replace: true } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false } } peers { permanent_uuid: "7505fefeaf804ed8aeb3672f386ab959" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 38645 } attrs { promote: false } }
W20240502 01:53:39.653263 13427 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:36164: Service unavailable: leader transfer in progress
I20240502 01:53:39.654703 13318 catalog_manager.cc:5484] T eb1cf83294e044b49e115323de67ed42 P 9e4fa889fab047baa5bd3dcd77b2ca8f reported cstate change: term changed from 1 to 2, leader changed from 5359629f948641f4b56e35ad6b6a15c4 (127.9.176.193) to 9e4fa889fab047baa5bd3dcd77b2ca8f (127.9.176.198). New cstate: current_term: 2 leader_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" committed_config { opid_index: 200 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7505fefeaf804ed8aeb3672f386ab959" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 38645 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240502 01:53:39.661031 13427 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:36168: Illegal state: replica 5359629f948641f4b56e35ad6b6a15c4 is not leader of this config: current role FOLLOWER
W20240502 01:53:39.663358 13427 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:36164: Service unavailable: leader transfer in progress
I20240502 01:53:39.664579 15228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb1cf83294e044b49e115323de67ed42" candidate_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" candidate_term: 2 candidate_status { last_received { term: 1 index: 206 } } ignore_live_leader: true dest_uuid: "7505fefeaf804ed8aeb3672f386ab959"
I20240502 01:53:39.664955 15228 raft_consensus.cc:3037] T eb1cf83294e044b49e115323de67ed42 P 7505fefeaf804ed8aeb3672f386ab959 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:53:39.665725 16203 tablet_replica.cc:330] T 930645182ab748d7b864fece1e858b3f P 9e4fa889fab047baa5bd3dcd77b2ca8f: stopping tablet replica
I20240502 01:53:39.666321 16203 raft_consensus.cc:2229] T 930645182ab748d7b864fece1e858b3f P 9e4fa889fab047baa5bd3dcd77b2ca8f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:39.666875 16203 raft_consensus.cc:2258] T 930645182ab748d7b864fece1e858b3f P 9e4fa889fab047baa5bd3dcd77b2ca8f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:39.668304 15228 raft_consensus.cc:2454] T eb1cf83294e044b49e115323de67ed42 P 7505fefeaf804ed8aeb3672f386ab959 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e4fa889fab047baa5bd3dcd77b2ca8f in term 2.
I20240502 01:53:39.669037 16203 ts_tablet_manager.cc:1875] T 930645182ab748d7b864fece1e858b3f P 9e4fa889fab047baa5bd3dcd77b2ca8f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240502 01:53:39.673310 13427 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:36164: Service unavailable: leader transfer in progress
I20240502 01:53:39.678339 16203 ts_tablet_manager.cc:1888] T 930645182ab748d7b864fece1e858b3f P 9e4fa889fab047baa5bd3dcd77b2ca8f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.185
I20240502 01:53:39.678656 16203 log.cc:1196] T 930645182ab748d7b864fece1e858b3f P 9e4fa889fab047baa5bd3dcd77b2ca8f: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/wal/wals/930645182ab748d7b864fece1e858b3f
I20240502 01:53:39.679895 13313 catalog_manager.cc:4830] TS 9e4fa889fab047baa5bd3dcd77b2ca8f (127.9.176.198:38527): tablet 930645182ab748d7b864fece1e858b3f (table rebalance_test_table_4 [id=7c97435ee22d42aaa3452b6e2abdaf75]) successfully deleted
W20240502 01:53:39.683416 13427 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:36164: Service unavailable: leader transfer in progress
W20240502 01:53:39.698468 13427 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:36164: Service unavailable: leader transfer in progress
W20240502 01:53:39.721632 13427 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:36164: Service unavailable: leader transfer in progress
W20240502 01:53:39.740221 13427 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:36164: Service unavailable: leader transfer in progress
W20240502 01:53:39.754964 13427 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:36164: Service unavailable: leader transfer in progress
I20240502 01:53:39.760974 15225 raft_consensus.cc:1261] T eb1cf83294e044b49e115323de67ed42 P 7505fefeaf804ed8aeb3672f386ab959 [term 2 FOLLOWER]: Refusing update from remote peer 9e4fa889fab047baa5bd3dcd77b2ca8f: Log matching property violated. Preceding OpId in replica: term: 1 index: 206. Preceding OpId from leader: term: 2 index: 208. (index mismatch)
I20240502 01:53:39.761713 16189 consensus_queue.cc:1035] T eb1cf83294e044b49e115323de67ed42 P 9e4fa889fab047baa5bd3dcd77b2ca8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7505fefeaf804ed8aeb3672f386ab959" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 38645 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 207, Last known committed idx: 206, Time since last communication: 0.000s
W20240502 01:53:39.772118 13427 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:36164: Service unavailable: leader transfer in progress
I20240502 01:53:39.773232 13470 raft_consensus.cc:1261] T eb1cf83294e044b49e115323de67ed42 P 5359629f948641f4b56e35ad6b6a15c4 [term 2 FOLLOWER]: Refusing update from remote peer 9e4fa889fab047baa5bd3dcd77b2ca8f: Log matching property violated. Preceding OpId in replica: term: 1 index: 206. Preceding OpId from leader: term: 2 index: 208. (index mismatch)
I20240502 01:53:39.780103 16189 consensus_queue.cc:1035] T eb1cf83294e044b49e115323de67ed42 P 9e4fa889fab047baa5bd3dcd77b2ca8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 207, Last known committed idx: 206, Time since last communication: 0.000s
I20240502 01:53:39.794433 16018 mvcc.cc:205] Tried to move back new op lower bound from 7023062301726846976 to 7023062301282652160. Current Snapshot: MvccSnapshot[applied={T|T < 7023062301726846976}]
W20240502 01:53:39.795979 13427 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:36164: Service unavailable: leader transfer in progress
I20240502 01:53:39.799211 16163 mvcc.cc:205] Tried to move back new op lower bound from 7023062301726846976 to 7023062301282652160. Current Snapshot: MvccSnapshot[applied={T|T < 7023062301726846976 or (T in {7023062301726846976})}]
I20240502 01:53:39.800930 15762 mvcc.cc:205] Tried to move back new op lower bound from 7023062301726846976 to 7023062301282652160. Current Snapshot: MvccSnapshot[applied={T|T < 7023062301726846976}]
I20240502 01:53:39.808151 15085 consensus_queue.cc:237] T eb1cf83294e044b49e115323de67ed42 P 9e4fa889fab047baa5bd3dcd77b2ca8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 208, Committed index: 208, Last appended: 2.208, Last appended by leader: 206, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 209 OBSOLETE_local: false peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false } } peers { permanent_uuid: "7505fefeaf804ed8aeb3672f386ab959" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 38645 } attrs { promote: false } }
I20240502 01:53:39.810472 15226 raft_consensus.cc:1261] T eb1cf83294e044b49e115323de67ed42 P 7505fefeaf804ed8aeb3672f386ab959 [term 2 FOLLOWER]: Refusing update from remote peer 9e4fa889fab047baa5bd3dcd77b2ca8f: Log matching property violated. Preceding OpId in replica: term: 2 index: 208. Preceding OpId from leader: term: 2 index: 209. (index mismatch)
I20240502 01:53:39.811174 16189 consensus_queue.cc:1035] T eb1cf83294e044b49e115323de67ed42 P 9e4fa889fab047baa5bd3dcd77b2ca8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7505fefeaf804ed8aeb3672f386ab959" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 38645 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 209, Last known committed idx: 208, Time since last communication: 0.000s
W20240502 01:53:39.815347 13426 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:36164: Service unavailable: leader transfer in progress
I20240502 01:53:39.816155 16189 raft_consensus.cc:2932] T eb1cf83294e044b49e115323de67ed42 P 9e4fa889fab047baa5bd3dcd77b2ca8f [term 2 LEADER]: Committing config change with OpId 2.209: config changed from index 200 to 209, VOTER 5359629f948641f4b56e35ad6b6a15c4 (127.9.176.193) evicted. New config: { opid_index: 209 OBSOLETE_local: false peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false } } peers { permanent_uuid: "7505fefeaf804ed8aeb3672f386ab959" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 38645 } attrs { promote: false } } }
I20240502 01:53:39.818017 15225 raft_consensus.cc:2932] T eb1cf83294e044b49e115323de67ed42 P 7505fefeaf804ed8aeb3672f386ab959 [term 2 FOLLOWER]: Committing config change with OpId 2.209: config changed from index 200 to 209, VOTER 5359629f948641f4b56e35ad6b6a15c4 (127.9.176.193) evicted. New config: { opid_index: 209 OBSOLETE_local: false peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false } } peers { permanent_uuid: "7505fefeaf804ed8aeb3672f386ab959" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 38645 } attrs { promote: false } } }
I20240502 01:53:39.821717 13313 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet eb1cf83294e044b49e115323de67ed42 with cas_config_opid_index 200: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:53:39.824757 13318 catalog_manager.cc:5484] T eb1cf83294e044b49e115323de67ed42 P 9e4fa889fab047baa5bd3dcd77b2ca8f reported cstate change: config changed from index 200 to 209, VOTER 5359629f948641f4b56e35ad6b6a15c4 (127.9.176.193) evicted. New cstate: current_term: 2 leader_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" committed_config { opid_index: 209 OBSOLETE_local: false peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7505fefeaf804ed8aeb3672f386ab959" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 38645 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:53:39.833973 13451 tablet_service.cc:1514] Processing DeleteTablet for tablet eb1cf83294e044b49e115323de67ed42 with delete_type TABLET_DATA_TOMBSTONED (TS 5359629f948641f4b56e35ad6b6a15c4 not found in new config with opid_index 209) from {username='slave'} at 127.0.0.1:38182
W20240502 01:53:39.841627 13427 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:36164: Service unavailable: leader transfer in progress
I20240502 01:53:39.852921 16221 tablet_replica.cc:330] T eb1cf83294e044b49e115323de67ed42 P 5359629f948641f4b56e35ad6b6a15c4: stopping tablet replica
I20240502 01:53:39.855376 16221 raft_consensus.cc:2229] T eb1cf83294e044b49e115323de67ed42 P 5359629f948641f4b56e35ad6b6a15c4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:39.855959 16221 raft_consensus.cc:2258] T eb1cf83294e044b49e115323de67ed42 P 5359629f948641f4b56e35ad6b6a15c4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:39.857508 16221 ts_tablet_manager.cc:1875] T eb1cf83294e044b49e115323de67ed42 P 5359629f948641f4b56e35ad6b6a15c4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240502 01:53:39.863597 13427 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:36164: Service unavailable: leader transfer in progress
I20240502 01:53:39.867100 15996 raft_consensus.cc:979] T b38a1fb4d1694c609a932e8816fc3514 P 5359629f948641f4b56e35ad6b6a15c4: : Instructing follower 7e74a3d5df6a483c95c9dabfc99577bc to start an election
I20240502 01:53:39.867295 15996 raft_consensus.cc:1067] T b38a1fb4d1694c609a932e8816fc3514 P 5359629f948641f4b56e35ad6b6a15c4 [term 1 LEADER]: Signalling peer 7e74a3d5df6a483c95c9dabfc99577bc to start an election
I20240502 01:53:39.868566 14954 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b38a1fb4d1694c609a932e8816fc3514"
dest_uuid: "7e74a3d5df6a483c95c9dabfc99577bc"
 from {username='slave'} at 127.9.176.193:53097
I20240502 01:53:39.868868 14954 raft_consensus.cc:482] T b38a1fb4d1694c609a932e8816fc3514 P 7e74a3d5df6a483c95c9dabfc99577bc [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:53:39.869046 14954 raft_consensus.cc:3037] T b38a1fb4d1694c609a932e8816fc3514 P 7e74a3d5df6a483c95c9dabfc99577bc [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:53:39.872061 14954 raft_consensus.cc:504] T b38a1fb4d1694c609a932e8816fc3514 P 7e74a3d5df6a483c95c9dabfc99577bc [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 177 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } attrs { replace: true } } peers { permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false } }
W20240502 01:53:39.886282 13427 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:36164: Service unavailable: leader transfer in progress
I20240502 01:53:39.887673 16221 ts_tablet_manager.cc:1888] T eb1cf83294e044b49e115323de67ed42 P 5359629f948641f4b56e35ad6b6a15c4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.208
I20240502 01:53:39.887938 16221 log.cc:1196] T eb1cf83294e044b49e115323de67ed42 P 5359629f948641f4b56e35ad6b6a15c4: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/wal/wals/eb1cf83294e044b49e115323de67ed42
I20240502 01:53:39.889925 13313 catalog_manager.cc:4830] TS 5359629f948641f4b56e35ad6b6a15c4 (127.9.176.193:33849): tablet eb1cf83294e044b49e115323de67ed42 (table rebalance_test_table_3 [id=c824fe62874b40458b3bf6ce96e3242b]) successfully deleted
I20240502 01:53:39.899888 13470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b38a1fb4d1694c609a932e8816fc3514" candidate_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" candidate_term: 2 candidate_status { last_received { term: 1 index: 185 } } ignore_live_leader: true dest_uuid: "5359629f948641f4b56e35ad6b6a15c4"
I20240502 01:53:39.900135 13470 raft_consensus.cc:3032] T b38a1fb4d1694c609a932e8816fc3514 P 5359629f948641f4b56e35ad6b6a15c4 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:53:39.900210 13470 raft_consensus.cc:729] T b38a1fb4d1694c609a932e8816fc3514 P 5359629f948641f4b56e35ad6b6a15c4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5359629f948641f4b56e35ad6b6a15c4, State: Running, Role: LEADER
I20240502 01:53:39.900491 13470 consensus_queue.cc:260] T b38a1fb4d1694c609a932e8816fc3514 P 5359629f948641f4b56e35ad6b6a15c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 185, Committed index: 185, Last appended: 1.185, Last appended by leader: 185, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 177 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } attrs { replace: true } } peers { permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false } }
I20240502 01:53:39.900866 13470 raft_consensus.cc:3037] T b38a1fb4d1694c609a932e8816fc3514 P 5359629f948641f4b56e35ad6b6a15c4 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:53:39.904516 13470 raft_consensus.cc:2454] T b38a1fb4d1694c609a932e8816fc3514 P 5359629f948641f4b56e35ad6b6a15c4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e74a3d5df6a483c95c9dabfc99577bc in term 2.
I20240502 01:53:39.905261 14890 leader_election.cc:302] T b38a1fb4d1694c609a932e8816fc3514 P 7e74a3d5df6a483c95c9dabfc99577bc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5359629f948641f4b56e35ad6b6a15c4, 7e74a3d5df6a483c95c9dabfc99577bc; no voters: 
I20240502 01:53:39.906004 16226 raft_consensus.cc:2781] T b38a1fb4d1694c609a932e8816fc3514 P 7e74a3d5df6a483c95c9dabfc99577bc [term 2 FOLLOWER]: Leader election won for term 2
W20240502 01:53:39.908473 13427 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:36164: Illegal state: replica 5359629f948641f4b56e35ad6b6a15c4 is not leader of this config: current role FOLLOWER
I20240502 01:53:39.908581 14954 leader_election.cc:288] T b38a1fb4d1694c609a932e8816fc3514 P 7e74a3d5df6a483c95c9dabfc99577bc [CANDIDATE]: Term 2 election: Requested vote from peers 5359629f948641f4b56e35ad6b6a15c4 (127.9.176.193:33849), 9e4fa889fab047baa5bd3dcd77b2ca8f (127.9.176.198:38527)
W20240502 01:53:39.960253 13427 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:36164: Illegal state: replica 5359629f948641f4b56e35ad6b6a15c4 is not leader of this config: current role FOLLOWER
I20240502 01:53:39.999483 15085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b38a1fb4d1694c609a932e8816fc3514" candidate_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" candidate_term: 2 candidate_status { last_received { term: 1 index: 185 } } ignore_live_leader: true dest_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f"
I20240502 01:53:39.999936 15085 raft_consensus.cc:3037] T b38a1fb4d1694c609a932e8816fc3514 P 9e4fa889fab047baa5bd3dcd77b2ca8f [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:53:40.000427 16226 raft_consensus.cc:686] T b38a1fb4d1694c609a932e8816fc3514 P 7e74a3d5df6a483c95c9dabfc99577bc [term 2 LEADER]: Becoming Leader. State: Replica: 7e74a3d5df6a483c95c9dabfc99577bc, State: Running, Role: LEADER
I20240502 01:53:40.001003 16226 consensus_queue.cc:237] T b38a1fb4d1694c609a932e8816fc3514 P 7e74a3d5df6a483c95c9dabfc99577bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 185, Committed index: 185, Last appended: 1.185, Last appended by leader: 185, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 177 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } attrs { replace: true } } peers { permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false } }
I20240502 01:53:40.003700 15085 raft_consensus.cc:2454] T b38a1fb4d1694c609a932e8816fc3514 P 9e4fa889fab047baa5bd3dcd77b2ca8f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e74a3d5df6a483c95c9dabfc99577bc in term 2.
I20240502 01:53:40.016885 13318 catalog_manager.cc:5484] T b38a1fb4d1694c609a932e8816fc3514 P 7e74a3d5df6a483c95c9dabfc99577bc reported cstate change: term changed from 1 to 2, leader changed from 5359629f948641f4b56e35ad6b6a15c4 (127.9.176.193) to 7e74a3d5df6a483c95c9dabfc99577bc (127.9.176.197). New cstate: current_term: 2 leader_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" committed_config { opid_index: 177 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:40.042380 15085 raft_consensus.cc:1261] T b38a1fb4d1694c609a932e8816fc3514 P 9e4fa889fab047baa5bd3dcd77b2ca8f [term 2 FOLLOWER]: Refusing update from remote peer 7e74a3d5df6a483c95c9dabfc99577bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 185. Preceding OpId from leader: term: 2 index: 187. (index mismatch)
I20240502 01:53:40.042929 13470 raft_consensus.cc:1261] T b38a1fb4d1694c609a932e8816fc3514 P 5359629f948641f4b56e35ad6b6a15c4 [term 2 FOLLOWER]: Refusing update from remote peer 7e74a3d5df6a483c95c9dabfc99577bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 185. Preceding OpId from leader: term: 2 index: 187. (index mismatch)
I20240502 01:53:40.043128 16226 consensus_queue.cc:1035] T b38a1fb4d1694c609a932e8816fc3514 P 7e74a3d5df6a483c95c9dabfc99577bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 186, Last known committed idx: 185, Time since last communication: 0.000s
I20240502 01:53:40.043764 16227 consensus_queue.cc:1035] T b38a1fb4d1694c609a932e8816fc3514 P 7e74a3d5df6a483c95c9dabfc99577bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 186, Last known committed idx: 185, Time since last communication: 0.000s
I20240502 01:53:40.046888 13470 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b38a1fb4d1694c609a932e8816fc3514"
dest_uuid: "5359629f948641f4b56e35ad6b6a15c4"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:38314
I20240502 01:53:40.047120 13470 raft_consensus.cc:595] T b38a1fb4d1694c609a932e8816fc3514 P 5359629f948641f4b56e35ad6b6a15c4 [term 2 FOLLOWER]: Received request to transfer leadership
I20240502 01:53:40.047235 13470 raft_consensus.cc:603] T b38a1fb4d1694c609a932e8816fc3514 P 5359629f948641f4b56e35ad6b6a15c4 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240502 01:53:40.048022 16225 mvcc.cc:205] Tried to move back new op lower bound from 7023062302885380096 to 7023062302764941312. Current Snapshot: MvccSnapshot[applied={T|T < 7023062302885380096}]
I20240502 01:53:40.074251 14953 consensus_queue.cc:237] T b38a1fb4d1694c609a932e8816fc3514 P 7e74a3d5df6a483c95c9dabfc99577bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 187, Committed index: 187, Last appended: 2.187, Last appended by leader: 185, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 188 OBSOLETE_local: false peers { permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false } }
I20240502 01:53:40.076680 15086 raft_consensus.cc:1261] T b38a1fb4d1694c609a932e8816fc3514 P 9e4fa889fab047baa5bd3dcd77b2ca8f [term 2 FOLLOWER]: Refusing update from remote peer 7e74a3d5df6a483c95c9dabfc99577bc: Log matching property violated. Preceding OpId in replica: term: 2 index: 187. Preceding OpId from leader: term: 2 index: 188. (index mismatch)
I20240502 01:53:40.077512 16226 consensus_queue.cc:1035] T b38a1fb4d1694c609a932e8816fc3514 P 7e74a3d5df6a483c95c9dabfc99577bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 188, Last known committed idx: 187, Time since last communication: 0.000s
I20240502 01:53:40.080849 16226 raft_consensus.cc:2932] T b38a1fb4d1694c609a932e8816fc3514 P 7e74a3d5df6a483c95c9dabfc99577bc [term 2 LEADER]: Committing config change with OpId 2.188: config changed from index 177 to 188, VOTER 5359629f948641f4b56e35ad6b6a15c4 (127.9.176.193) evicted. New config: { opid_index: 188 OBSOLETE_local: false peers { permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false } } }
I20240502 01:53:40.084327 15086 raft_consensus.cc:2932] T b38a1fb4d1694c609a932e8816fc3514 P 9e4fa889fab047baa5bd3dcd77b2ca8f [term 2 FOLLOWER]: Committing config change with OpId 2.188: config changed from index 177 to 188, VOTER 5359629f948641f4b56e35ad6b6a15c4 (127.9.176.193) evicted. New config: { opid_index: 188 OBSOLETE_local: false peers { permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false } } }
I20240502 01:53:40.086947 13310 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b38a1fb4d1694c609a932e8816fc3514 with cas_config_opid_index 177: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:53:40.089578 13318 catalog_manager.cc:5484] T b38a1fb4d1694c609a932e8816fc3514 P 7e74a3d5df6a483c95c9dabfc99577bc reported cstate change: config changed from index 177 to 188, VOTER 5359629f948641f4b56e35ad6b6a15c4 (127.9.176.193) evicted. New cstate: current_term: 2 leader_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" committed_config { opid_index: 188 OBSOLETE_local: false peers { permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:53:40.095075 13451 tablet_service.cc:1514] Processing DeleteTablet for tablet b38a1fb4d1694c609a932e8816fc3514 with delete_type TABLET_DATA_TOMBSTONED (TS 5359629f948641f4b56e35ad6b6a15c4 not found in new config with opid_index 188) from {username='slave'} at 127.0.0.1:38182
I20240502 01:53:40.095746 16221 tablet_replica.cc:330] T b38a1fb4d1694c609a932e8816fc3514 P 5359629f948641f4b56e35ad6b6a15c4: stopping tablet replica
I20240502 01:53:40.096011 16221 raft_consensus.cc:2229] T b38a1fb4d1694c609a932e8816fc3514 P 5359629f948641f4b56e35ad6b6a15c4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:40.096304 16221 raft_consensus.cc:2258] T b38a1fb4d1694c609a932e8816fc3514 P 5359629f948641f4b56e35ad6b6a15c4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:40.097721 16221 ts_tablet_manager.cc:1875] T b38a1fb4d1694c609a932e8816fc3514 P 5359629f948641f4b56e35ad6b6a15c4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:53:40.106027 16221 ts_tablet_manager.cc:1888] T b38a1fb4d1694c609a932e8816fc3514 P 5359629f948641f4b56e35ad6b6a15c4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.187
I20240502 01:53:40.106308 16221 log.cc:1196] T b38a1fb4d1694c609a932e8816fc3514 P 5359629f948641f4b56e35ad6b6a15c4: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/wal/wals/b38a1fb4d1694c609a932e8816fc3514
I20240502 01:53:40.111990 13313 catalog_manager.cc:4830] TS 5359629f948641f4b56e35ad6b6a15c4 (127.9.176.193:33849): tablet b38a1fb4d1694c609a932e8816fc3514 (table rebalance_test_table_1 [id=8ebdfe08a0b244819813a774f9d721fc]) successfully deleted
I20240502 01:53:41.007169 13584 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:53:41.097362 14799 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:41.104796 13451 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:53:41.105726 15067 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:41.108234 14934 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:41.115896 13715 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:53:41.143080 15208 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:42.195819 15208 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:42.232681 13715 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:53:42.250283 13451 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:53:42.252682 15067 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:42.262014 13584 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:53:42.292939 14799 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:42.300473 14934 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:53:43.920665 13671 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.176.199:38645, user_credentials={real_user=slave}} blocked reactor thread for 45415.4us
I20240502 01:53:44.144280 15208 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:44.219991 13451 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:53:44.229319 13584 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:53:44.259794 14934 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:44.367282 13715 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:53:44.419188 15067 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:44.475163 14799 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:45.546998 13451 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:53:45.620643 15208 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:45.651618 15067 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:45.682668 13715 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:53:45.708124 14934 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:45.745990 14799 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:45.755617 13584 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:53:46.440948 14934 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:46.458261 13451 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:53:46.468130 15208 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:46.474507 13584 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:53:46.509893 15067 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:46.524533 13715 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:53:46.530115 14799 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
               UUID               |       Address       | Status
----------------------------------+---------------------+---------
 fad8aa713de7433db0f21791b371a3e4 | 127.9.176.254:44447 | HEALTHY

Unusual flags for Master:
                   Flag                   |                                                                                Value                                                                                |         Tags          |         Master
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
 allow_unsafe_replication_factor          | true                                                                                                                                                                | runtime,unsafe        | all 1 server(s) checked
 ipki_ca_key_size                         | 768                                                                                                                                                                 | experimental          | all 1 server(s) checked
 ipki_server_key_size                     | 768                                                                                                                                                                 | experimental          | all 1 server(s) checked
 never_fsync                              | true                                                                                                                                                                | unsafe,advanced       | all 1 server(s) checked
 openssl_security_level_override          | 0                                                                                                                                                                   | unsafe,hidden         | all 1 server(s) checked
 raft_prepare_replacement_before_eviction | true                                                                                                                                                                | experimental,advanced | all 1 server(s) checked
 rpc_reuseport                            | true                                                                                                                                                                | experimental          | all 1 server(s) checked
 rpc_server_allow_ephemeral_ports         | true                                                                                                                                                                | unsafe                | all 1 server(s) checked
 server_dump_info_format                  | pb                                                                                                                                                                  | hidden                | all 1 server(s) checked
 server_dump_info_path                    | /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden                | all 1 server(s) checked
 tsk_num_rsa_bits                         | 512                                                                                                                                                                 | experimental          | all 1 server(s) checked

Flags of checked categories for Master:
        Flag         |        Value        |         Master
---------------------+---------------------+-------------------------
 builtin_ntp_servers | 127.9.176.212:33337 | all 1 server(s) checked
 time_source         | builtin             | all 1 server(s) checked

Tablet Server Summary
               UUID               |       Address       | Status  | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
 5359629f948641f4b56e35ad6b6a15c4 | 127.9.176.193:33849 | HEALTHY | <none>   |       5        |       0
 7505fefeaf804ed8aeb3672f386ab959 | 127.9.176.199:38645 | HEALTHY | <none>   |       0        |       0
 7e74a3d5df6a483c95c9dabfc99577bc | 127.9.176.197:35757 | HEALTHY | <none>   |       1        |       0
 8446b7a7ea67486288d174eea7a3b3c6 | 127.9.176.194:34851 | HEALTHY | <none>   |       3        |       0
 9e4fa889fab047baa5bd3dcd77b2ca8f | 127.9.176.198:38527 | HEALTHY | <none>   |       1        |       0
 b4abc88ed2e54830ae3a317629916117 | 127.9.176.195:41211 | HEALTHY | <none>   |       5        |       0
 ce90a4bd06ba41dd8e5aa227de9780a5 | 127.9.176.196:41329 | HEALTHY | <none>   |       1        |       0

Tablet Server Location Summary
 Location |  Count
----------+---------
 <none>   |       7

Unusual flags for Tablet Server:
                   Flag                   |                                                                              Value                                                                              |         Tags          |      Tablet Server
------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
 ipki_server_key_size                     | 768                                                                                                                                                             | experimental          | all 7 server(s) checked
 local_ip_for_outbound_sockets            | 127.9.176.193                                                                                                                                                   | experimental          | 127.9.176.193:33849
 local_ip_for_outbound_sockets            | 127.9.176.194                                                                                                                                                   | experimental          | 127.9.176.194:34851
 local_ip_for_outbound_sockets            | 127.9.176.195                                                                                                                                                   | experimental          | 127.9.176.195:41211
 local_ip_for_outbound_sockets            | 127.9.176.196                                                                                                                                                   | experimental          | 127.9.176.196:41329
 local_ip_for_outbound_sockets            | 127.9.176.197                                                                                                                                                   | experimental          | 127.9.176.197:35757
 local_ip_for_outbound_sockets            | 127.9.176.198                                                                                                                                                   | experimental          | 127.9.176.198:38527
 local_ip_for_outbound_sockets            | 127.9.176.199                                                                                                                                                   | experimental          | 127.9.176.199:38645
 never_fsync                              | true                                                                                                                                                            | unsafe,advanced       | all 7 server(s) checked
 openssl_security_level_override          | 0                                                                                                                                                               | unsafe,hidden         | all 7 server(s) checked
 raft_prepare_replacement_before_eviction | true                                                                                                                                                            | experimental,advanced | all 7 server(s) checked
 rpc_server_allow_ephemeral_ports         | true                                                                                                                                                            | unsafe                | all 7 server(s) checked
 server_dump_info_format                  | pb                                                                                                                                                              | hidden                | all 7 server(s) checked
 server_dump_info_path                    | /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden                | 127.9.176.193:33849
 server_dump_info_path                    | /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden                | 127.9.176.194:34851
 server_dump_info_path                    | /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden                | 127.9.176.195:41211
 server_dump_info_path                    | /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden                | 127.9.176.196:41329
 server_dump_info_path                    | /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden                | 127.9.176.197:35757
 server_dump_info_path                    | /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden                | 127.9.176.198:38527
 server_dump_info_path                    | /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden                | 127.9.176.199:38645

Flags of checked categories for Tablet Server:
        Flag         |        Value        |      Tablet Server
---------------------+---------------------+-------------------------
 builtin_ntp_servers | 127.9.176.212:33337 | all 7 server(s) checked
 time_source         | builtin             | all 7 server(s) checked

Version Summary
     Version     |         Servers
-----------------+-------------------------
 1.18.0-SNAPSHOT | all 8 server(s) checked

Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
          Name          | RF | Status  | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
 TestTable              | 2  | HEALTHY | 1             | 1       | 0          | 0                | 0
 rebalance_test_table_0 | 2  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_1 | 2  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_2 | 2  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_3 | 2  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_4 | 2  | HEALTHY | 3             | 3       | 0          | 0                | 0

Tablet Replica Count Summary
   Statistic    | Replica Count
----------------+---------------
 Minimum        | 4
 First Quartile | 4
 Median         | 5
 Third Quartile | 5
 Maximum        | 5

Total Count Summary
                | Total Count
----------------+-------------
 Masters        | 1
 Tablet Servers | 7
 Tables         | 6
 Tablets        | 16
 Replicas       | 32

==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set

OK
I20240502 01:53:46.602576  9923 log_verifier.cc:126] Checking tablet 01fb93c5dab3401fa5a103ead4dd9708
I20240502 01:53:46.738688  9923 log_verifier.cc:177] Verified matching terms for 349 ops in tablet 01fb93c5dab3401fa5a103ead4dd9708
I20240502 01:53:46.739240  9923 log_verifier.cc:126] Checking tablet 4c0ab7c2995d4ccfa170f365b4e09b1d
I20240502 01:53:46.879498  9923 log_verifier.cc:177] Verified matching terms for 332 ops in tablet 4c0ab7c2995d4ccfa170f365b4e09b1d
I20240502 01:53:46.880017  9923 log_verifier.cc:126] Checking tablet 57765e8e3ff7438ca9e588db8e685a03
I20240502 01:53:47.059635  9923 log_verifier.cc:177] Verified matching terms for 339 ops in tablet 57765e8e3ff7438ca9e588db8e685a03
I20240502 01:53:47.060314  9923 log_verifier.cc:126] Checking tablet 6682ed5cba8448c7b3cf74ab726a18e7
I20240502 01:53:47.255949  9923 log_verifier.cc:177] Verified matching terms for 303 ops in tablet 6682ed5cba8448c7b3cf74ab726a18e7
I20240502 01:53:47.256500  9923 log_verifier.cc:126] Checking tablet 71b0f9fbee82464994252126b2d631bc
I20240502 01:53:47.449457  9923 log_verifier.cc:177] Verified matching terms for 297 ops in tablet 71b0f9fbee82464994252126b2d631bc
I20240502 01:53:47.449978  9923 log_verifier.cc:126] Checking tablet 8477215e8d4043c581c4ce7e5aa0fbd6
I20240502 01:53:47.606036  9923 log_verifier.cc:177] Verified matching terms for 293 ops in tablet 8477215e8d4043c581c4ce7e5aa0fbd6
I20240502 01:53:47.606451  9923 log_verifier.cc:126] Checking tablet 866d771c178a450e84189c64ddfca9d7
I20240502 01:53:47.756287  9923 log_verifier.cc:177] Verified matching terms for 298 ops in tablet 866d771c178a450e84189c64ddfca9d7
I20240502 01:53:47.756718  9923 log_verifier.cc:126] Checking tablet 930645182ab748d7b864fece1e858b3f
I20240502 01:53:47.925563  9923 log_verifier.cc:177] Verified matching terms for 294 ops in tablet 930645182ab748d7b864fece1e858b3f
I20240502 01:53:47.925985  9923 log_verifier.cc:126] Checking tablet b38a1fb4d1694c609a932e8816fc3514
I20240502 01:53:48.135314  9923 log_verifier.cc:177] Verified matching terms for 315 ops in tablet b38a1fb4d1694c609a932e8816fc3514
I20240502 01:53:48.135957  9923 log_verifier.cc:126] Checking tablet bdf6095f84aa497ba5db95d55c010085
I20240502 01:53:48.322278  9923 log_verifier.cc:177] Verified matching terms for 311 ops in tablet bdf6095f84aa497ba5db95d55c010085
I20240502 01:53:48.322758  9923 log_verifier.cc:126] Checking tablet d38a05f7ffa94cc5abc50a76a83692b5
I20240502 01:53:48.392031  9923 log_verifier.cc:177] Verified matching terms for 5 ops in tablet d38a05f7ffa94cc5abc50a76a83692b5
I20240502 01:53:48.392145  9923 log_verifier.cc:126] Checking tablet d4511dd1019c4bdcbf9fc7e4bfe2afb7
I20240502 01:53:48.514719  9923 log_verifier.cc:177] Verified matching terms for 308 ops in tablet d4511dd1019c4bdcbf9fc7e4bfe2afb7
I20240502 01:53:48.515153  9923 log_verifier.cc:126] Checking tablet daed3123a96a496da83a6110e6943264
I20240502 01:53:48.671988  9923 log_verifier.cc:177] Verified matching terms for 329 ops in tablet daed3123a96a496da83a6110e6943264
I20240502 01:53:48.672448  9923 log_verifier.cc:126] Checking tablet eb1cf83294e044b49e115323de67ed42
I20240502 01:53:48.834573  9923 log_verifier.cc:177] Verified matching terms for 324 ops in tablet eb1cf83294e044b49e115323de67ed42
I20240502 01:53:48.835026  9923 log_verifier.cc:126] Checking tablet edf0aa94bfb94d499015897713ed94fa
I20240502 01:53:49.001487  9923 log_verifier.cc:177] Verified matching terms for 306 ops in tablet edf0aa94bfb94d499015897713ed94fa
I20240502 01:53:49.002085  9923 log_verifier.cc:126] Checking tablet f2af8f5d5e784e1988a957666844db6b
I20240502 01:53:49.169503  9923 log_verifier.cc:177] Verified matching terms for 294 ops in tablet f2af8f5d5e784e1988a957666844db6b
I20240502 01:53:49.173238  9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
/tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.200:0
--local_ip_for_outbound_sockets=127.9.176.200
--webserver_interface=127.9.176.200
--webserver_port=0
--tserver_master_addrs=127.9.176.254:44447
--builtin_ntp_servers=127.9.176.212:33337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 01:53:49.600242 16552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:49.600543 16552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:53:49.600692 16552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:49.609234 16552 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:53:49.609395 16552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:53:49.609567 16552 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.200
I20240502 01:53:49.619966 16552 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:33337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.200:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.9.176.200
--webserver_port=0
--tserver_master_addrs=127.9.176.254:44447
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.200
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:53:49.621448 16552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:53:49.623406 16552 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:53:49.640416 16560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:49.640930 16559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:49.641422 16562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:53:49.642194 16552 server_base.cc:1034] running on GCE node
I20240502 01:53:49.642874 16552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:53:49.644016 16552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:53:49.645246 16552 hybrid_clock.cc:648] HybridClock initialized: now 1714614829645189 us; error 61 us; skew 500 ppm
I20240502 01:53:49.645606 16552 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:53:49.654350 16552 webserver.cc:460] Webserver started at http://127.9.176.200:33685/ using document root <none> and password file <none>
I20240502 01:53:49.655295 16552 fs_manager.cc:359] Metadata directory not provided
I20240502 01:53:49.655428 16552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:53:49.655812 16552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:53:49.659071 16552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "a82548707a5844ce9650f2e9dbe4f581"
format_stamp: "Formatted at 2024-05-02 01:53:49 on dist-test-slave-j3hh"
I20240502 01:53:49.660359 16552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "a82548707a5844ce9650f2e9dbe4f581"
format_stamp: "Formatted at 2024-05-02 01:53:49 on dist-test-slave-j3hh"
I20240502 01:53:49.667060 16552 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.005s	sys 0.001s
I20240502 01:53:49.671463 16568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:53:49.673223 16552 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.000s
I20240502 01:53:49.673456 16552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "a82548707a5844ce9650f2e9dbe4f581"
format_stamp: "Formatted at 2024-05-02 01:53:49 on dist-test-slave-j3hh"
I20240502 01:53:49.673703 16552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:53:49.691385 16552 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:53:49.692422 16552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:53:49.692772 16552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:53:49.693969 16552 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:53:49.695787 16552 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:53:49.695966 16552 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:53:49.696098 16552 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:53:49.696180 16552 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:53:49.739518 16552 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.200:42345
I20240502 01:53:49.739662 16682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.200:42345 every 8 connection(s)
I20240502 01:53:49.741111 16552 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 01:53:49.749132  9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu as pid 16552
I20240502 01:53:49.749383  9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240502 01:53:49.757853 16683 heartbeater.cc:347] Connected to a master server at 127.9.176.254:44447
I20240502 01:53:49.758324 16683 heartbeater.cc:464] Registering TS with master...
I20240502 01:53:49.759876 16683 heartbeater.cc:511] Master 127.9.176.254:44447 requested a full tablet report, sending...
I20240502 01:53:49.762071 13325 ts_manager.cc:194] Registered new tserver with Master: a82548707a5844ce9650f2e9dbe4f581 (127.9.176.200:42345)
I20240502 01:53:49.763298 13325 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.176.200:46041
I20240502 01:53:50.370692 14934 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:50.383488 13584 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:53:50.402417 13715 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:53:50.412318 15208 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:50.412533 13451 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:53:50.423228 15067 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:50.428493 16617 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:50.477536 14799 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:50.679685 13451 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:53:50.681219 14934 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:50.681466 15208 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:50.686345 13715 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:53:50.686738 16617 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:50.687130 15067 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:50.708170 14799 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:50.716240 13584 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:53:50.766728 16683 heartbeater.cc:503] Master 127.9.176.254:44447 was elected leader, sending a full tablet report...
I20240502 01:53:50.842483 13470 consensus_queue.cc:237] T 6682ed5cba8448c7b3cf74ab726a18e7 P 5359629f948641f4b56e35ad6b6a15c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 303, Committed index: 303, Last appended: 1.303, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 304 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } attrs { replace: true } } peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "a82548707a5844ce9650f2e9dbe4f581" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 42345 } attrs { promote: true } }
I20240502 01:53:50.845820 13604 raft_consensus.cc:1261] T 6682ed5cba8448c7b3cf74ab726a18e7 P 8446b7a7ea67486288d174eea7a3b3c6 [term 1 FOLLOWER]: Refusing update from remote peer 5359629f948641f4b56e35ad6b6a15c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 303. Preceding OpId from leader: term: 1 index: 304. (index mismatch)
I20240502 01:53:50.846546 16779 consensus_queue.cc:1035] T 6682ed5cba8448c7b3cf74ab726a18e7 P 5359629f948641f4b56e35ad6b6a15c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 304, Last known committed idx: 303, Time since last communication: 0.000s
I20240502 01:53:50.850045 16467 raft_consensus.cc:2932] T 6682ed5cba8448c7b3cf74ab726a18e7 P 5359629f948641f4b56e35ad6b6a15c4 [term 1 LEADER]: Committing config change with OpId 1.304: config changed from index -1 to 304, NON_VOTER a82548707a5844ce9650f2e9dbe4f581 (127.9.176.200) added. New config: { opid_index: 304 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } attrs { replace: true } } peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "a82548707a5844ce9650f2e9dbe4f581" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 42345 } attrs { promote: true } } }
I20240502 01:53:50.850780 13604 raft_consensus.cc:2932] T 6682ed5cba8448c7b3cf74ab726a18e7 P 8446b7a7ea67486288d174eea7a3b3c6 [term 1 FOLLOWER]: Committing config change with OpId 1.304: config changed from index -1 to 304, NON_VOTER a82548707a5844ce9650f2e9dbe4f581 (127.9.176.200) added. New config: { opid_index: 304 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } attrs { replace: true } } peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "a82548707a5844ce9650f2e9dbe4f581" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 42345 } attrs { promote: true } } }
W20240502 01:53:50.853462 13405 consensus_peers.cc:485] T 6682ed5cba8448c7b3cf74ab726a18e7 P 5359629f948641f4b56e35ad6b6a15c4 -> Peer a82548707a5844ce9650f2e9dbe4f581 (127.9.176.200:42345): Couldn't send request to peer a82548707a5844ce9650f2e9dbe4f581. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6682ed5cba8448c7b3cf74ab726a18e7. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:50.855831 13319 catalog_manager.cc:5484] T 6682ed5cba8448c7b3cf74ab726a18e7 P 5359629f948641f4b56e35ad6b6a15c4 reported cstate change: config changed from index -1 to 304, NON_VOTER a82548707a5844ce9650f2e9dbe4f581 (127.9.176.200) added. New cstate: current_term: 1 leader_uuid: "5359629f948641f4b56e35ad6b6a15c4" committed_config { opid_index: 304 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a82548707a5844ce9650f2e9dbe4f581" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 42345 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:50.869349 13470 consensus_queue.cc:237] T bdf6095f84aa497ba5db95d55c010085 P 5359629f948641f4b56e35ad6b6a15c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 311, Committed index: 311, Last appended: 1.311, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 312 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } attrs { replace: true } } peers { permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 41329 } attrs { promote: false } } peers { permanent_uuid: "a82548707a5844ce9650f2e9dbe4f581" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 42345 } attrs { promote: true } }
I20240502 01:53:50.872033 14820 raft_consensus.cc:1261] T bdf6095f84aa497ba5db95d55c010085 P ce90a4bd06ba41dd8e5aa227de9780a5 [term 1 FOLLOWER]: Refusing update from remote peer 5359629f948641f4b56e35ad6b6a15c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 311. Preceding OpId from leader: term: 1 index: 312. (index mismatch)
W20240502 01:53:50.872246 13405 consensus_peers.cc:485] T bdf6095f84aa497ba5db95d55c010085 P 5359629f948641f4b56e35ad6b6a15c4 -> Peer a82548707a5844ce9650f2e9dbe4f581 (127.9.176.200:42345): Couldn't send request to peer a82548707a5844ce9650f2e9dbe4f581. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bdf6095f84aa497ba5db95d55c010085. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:50.872690 16467 consensus_queue.cc:1035] T bdf6095f84aa497ba5db95d55c010085 P 5359629f948641f4b56e35ad6b6a15c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 41329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 312, Last known committed idx: 311, Time since last communication: 0.000s
I20240502 01:53:50.876580 16779 raft_consensus.cc:2932] T bdf6095f84aa497ba5db95d55c010085 P 5359629f948641f4b56e35ad6b6a15c4 [term 1 LEADER]: Committing config change with OpId 1.312: config changed from index 94 to 312, NON_VOTER a82548707a5844ce9650f2e9dbe4f581 (127.9.176.200) added. New config: { opid_index: 312 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } attrs { replace: true } } peers { permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 41329 } attrs { promote: false } } peers { permanent_uuid: "a82548707a5844ce9650f2e9dbe4f581" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 42345 } attrs { promote: true } } }
I20240502 01:53:50.877848 14820 raft_consensus.cc:2932] T bdf6095f84aa497ba5db95d55c010085 P ce90a4bd06ba41dd8e5aa227de9780a5 [term 1 FOLLOWER]: Committing config change with OpId 1.312: config changed from index 94 to 312, NON_VOTER a82548707a5844ce9650f2e9dbe4f581 (127.9.176.200) added. New config: { opid_index: 312 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } attrs { replace: true } } peers { permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 41329 } attrs { promote: false } } peers { permanent_uuid: "a82548707a5844ce9650f2e9dbe4f581" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 42345 } attrs { promote: true } } }
I20240502 01:53:50.883595 13317 catalog_manager.cc:5484] T bdf6095f84aa497ba5db95d55c010085 P 5359629f948641f4b56e35ad6b6a15c4 reported cstate change: config changed from index 94 to 312, NON_VOTER a82548707a5844ce9650f2e9dbe4f581 (127.9.176.200) added. New cstate: current_term: 1 leader_uuid: "5359629f948641f4b56e35ad6b6a15c4" committed_config { opid_index: 312 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 41329 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a82548707a5844ce9650f2e9dbe4f581" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 42345 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:50.908396 13732 consensus_queue.cc:237] T f2af8f5d5e784e1988a957666844db6b P b4abc88ed2e54830ae3a317629916117 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 294, Committed index: 294, Last appended: 1.294, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 295 OBSOLETE_local: false peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "7505fefeaf804ed8aeb3672f386ab959" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 38645 } attrs { promote: false } } peers { permanent_uuid: "a82548707a5844ce9650f2e9dbe4f581" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 42345 } attrs { promote: true } }
I20240502 01:53:50.911912 15226 raft_consensus.cc:1261] T f2af8f5d5e784e1988a957666844db6b P 7505fefeaf804ed8aeb3672f386ab959 [term 1 FOLLOWER]: Refusing update from remote peer b4abc88ed2e54830ae3a317629916117: Log matching property violated. Preceding OpId in replica: term: 1 index: 294. Preceding OpId from leader: term: 1 index: 295. (index mismatch)
I20240502 01:53:50.912667 16414 consensus_queue.cc:1035] T f2af8f5d5e784e1988a957666844db6b P b4abc88ed2e54830ae3a317629916117 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7505fefeaf804ed8aeb3672f386ab959" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 38645 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 295, Last known committed idx: 294, Time since last communication: 0.000s
I20240502 01:53:50.916144 16800 raft_consensus.cc:2932] T f2af8f5d5e784e1988a957666844db6b P b4abc88ed2e54830ae3a317629916117 [term 1 LEADER]: Committing config change with OpId 1.295: config changed from index 78 to 295, NON_VOTER a82548707a5844ce9650f2e9dbe4f581 (127.9.176.200) added. New config: { opid_index: 295 OBSOLETE_local: false peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "7505fefeaf804ed8aeb3672f386ab959" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 38645 } attrs { promote: false } } peers { permanent_uuid: "a82548707a5844ce9650f2e9dbe4f581" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 42345 } attrs { promote: true } } }
I20240502 01:53:50.916965 15226 raft_consensus.cc:2932] T f2af8f5d5e784e1988a957666844db6b P 7505fefeaf804ed8aeb3672f386ab959 [term 1 FOLLOWER]: Committing config change with OpId 1.295: config changed from index 78 to 295, NON_VOTER a82548707a5844ce9650f2e9dbe4f581 (127.9.176.200) added. New config: { opid_index: 295 OBSOLETE_local: false peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "7505fefeaf804ed8aeb3672f386ab959" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 38645 } attrs { promote: false } } peers { permanent_uuid: "a82548707a5844ce9650f2e9dbe4f581" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 42345 } attrs { promote: true } } }
W20240502 01:53:50.918872 13671 consensus_peers.cc:485] T f2af8f5d5e784e1988a957666844db6b P b4abc88ed2e54830ae3a317629916117 -> Peer a82548707a5844ce9650f2e9dbe4f581 (127.9.176.200:42345): Couldn't send request to peer a82548707a5844ce9650f2e9dbe4f581. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f2af8f5d5e784e1988a957666844db6b. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:50.921566 13319 catalog_manager.cc:5484] T f2af8f5d5e784e1988a957666844db6b P b4abc88ed2e54830ae3a317629916117 reported cstate change: config changed from index 78 to 295, NON_VOTER a82548707a5844ce9650f2e9dbe4f581 (127.9.176.200) added. New cstate: current_term: 1 leader_uuid: "b4abc88ed2e54830ae3a317629916117" committed_config { opid_index: 295 OBSOLETE_local: false peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7505fefeaf804ed8aeb3672f386ab959" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 38645 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a82548707a5844ce9650f2e9dbe4f581" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 42345 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:50.943854 14954 consensus_queue.cc:237] T b38a1fb4d1694c609a932e8816fc3514 P 7e74a3d5df6a483c95c9dabfc99577bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 315, Committed index: 315, Last appended: 2.315, Last appended by leader: 185, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 316 OBSOLETE_local: false peers { permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false replace: true } } peers { permanent_uuid: "a82548707a5844ce9650f2e9dbe4f581" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 42345 } attrs { promote: true } }
I20240502 01:53:50.947808 15085 raft_consensus.cc:1261] T b38a1fb4d1694c609a932e8816fc3514 P 9e4fa889fab047baa5bd3dcd77b2ca8f [term 2 FOLLOWER]: Refusing update from remote peer 7e74a3d5df6a483c95c9dabfc99577bc: Log matching property violated. Preceding OpId in replica: term: 2 index: 315. Preceding OpId from leader: term: 2 index: 316. (index mismatch)
I20240502 01:53:50.948654 16689 consensus_queue.cc:1035] T b38a1fb4d1694c609a932e8816fc3514 P 7e74a3d5df6a483c95c9dabfc99577bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 316, Last known committed idx: 315, Time since last communication: 0.000s
W20240502 01:53:50.957811 14890 consensus_peers.cc:485] T b38a1fb4d1694c609a932e8816fc3514 P 7e74a3d5df6a483c95c9dabfc99577bc -> Peer a82548707a5844ce9650f2e9dbe4f581 (127.9.176.200:42345): Couldn't send request to peer a82548707a5844ce9650f2e9dbe4f581. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b38a1fb4d1694c609a932e8816fc3514. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:50.964689 16689 raft_consensus.cc:2932] T b38a1fb4d1694c609a932e8816fc3514 P 7e74a3d5df6a483c95c9dabfc99577bc [term 2 LEADER]: Committing config change with OpId 2.316: config changed from index 188 to 316, NON_VOTER a82548707a5844ce9650f2e9dbe4f581 (127.9.176.200) added. New config: { opid_index: 316 OBSOLETE_local: false peers { permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false replace: true } } peers { permanent_uuid: "a82548707a5844ce9650f2e9dbe4f581" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 42345 } attrs { promote: true } } }
I20240502 01:53:50.965629 15085 raft_consensus.cc:2932] T b38a1fb4d1694c609a932e8816fc3514 P 9e4fa889fab047baa5bd3dcd77b2ca8f [term 2 FOLLOWER]: Committing config change with OpId 2.316: config changed from index 188 to 316, NON_VOTER a82548707a5844ce9650f2e9dbe4f581 (127.9.176.200) added. New config: { opid_index: 316 OBSOLETE_local: false peers { permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false replace: true } } peers { permanent_uuid: "a82548707a5844ce9650f2e9dbe4f581" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 42345 } attrs { promote: true } } }
I20240502 01:53:50.971477 13319 catalog_manager.cc:5484] T b38a1fb4d1694c609a932e8816fc3514 P 9e4fa889fab047baa5bd3dcd77b2ca8f reported cstate change: config changed from index 188 to 316, NON_VOTER a82548707a5844ce9650f2e9dbe4f581 (127.9.176.200) added. New cstate: current_term: 2 leader_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" committed_config { opid_index: 316 OBSOLETE_local: false peers { permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false replace: true } } peers { permanent_uuid: "a82548707a5844ce9650f2e9dbe4f581" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 42345 } attrs { promote: true } } }
I20240502 01:53:51.282109 16857 ts_tablet_manager.cc:902] T bdf6095f84aa497ba5db95d55c010085 P a82548707a5844ce9650f2e9dbe4f581: Initiating tablet copy from peer 5359629f948641f4b56e35ad6b6a15c4 (127.9.176.193:33849)
I20240502 01:53:51.284191 16857 tablet_copy_client.cc:308] T bdf6095f84aa497ba5db95d55c010085 P a82548707a5844ce9650f2e9dbe4f581: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.193:33849
I20240502 01:53:51.291816 13491 tablet_copy_service.cc:140] P 5359629f948641f4b56e35ad6b6a15c4: Received BeginTabletCopySession request for tablet bdf6095f84aa497ba5db95d55c010085 from peer a82548707a5844ce9650f2e9dbe4f581 ({username='slave'} at 127.9.176.200:39765)
I20240502 01:53:51.292028 13491 tablet_copy_service.cc:161] P 5359629f948641f4b56e35ad6b6a15c4: Beginning new tablet copy session on tablet bdf6095f84aa497ba5db95d55c010085 from peer a82548707a5844ce9650f2e9dbe4f581 at {username='slave'} at 127.9.176.200:39765: session id = a82548707a5844ce9650f2e9dbe4f581-bdf6095f84aa497ba5db95d55c010085
I20240502 01:53:51.294378 13491 tablet_copy_source_session.cc:205] T bdf6095f84aa497ba5db95d55c010085 P 5359629f948641f4b56e35ad6b6a15c4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:51.299099 16857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdf6095f84aa497ba5db95d55c010085. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:51.310496 16857 tablet_copy_client.cc:783] T bdf6095f84aa497ba5db95d55c010085 P a82548707a5844ce9650f2e9dbe4f581: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:51.310946 16857 tablet_copy_client.cc:647] T bdf6095f84aa497ba5db95d55c010085 P a82548707a5844ce9650f2e9dbe4f581: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:51.314481 16857 tablet_copy_client.cc:523] T bdf6095f84aa497ba5db95d55c010085 P a82548707a5844ce9650f2e9dbe4f581: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:51.317941 16857 tablet_bootstrap.cc:492] T bdf6095f84aa497ba5db95d55c010085 P a82548707a5844ce9650f2e9dbe4f581: Bootstrap starting.
I20240502 01:53:51.343995 16863 ts_tablet_manager.cc:902] T b38a1fb4d1694c609a932e8816fc3514 P a82548707a5844ce9650f2e9dbe4f581: Initiating tablet copy from peer 7e74a3d5df6a483c95c9dabfc99577bc (127.9.176.197:35757)
I20240502 01:53:51.346078 16863 tablet_copy_client.cc:308] T b38a1fb4d1694c609a932e8816fc3514 P a82548707a5844ce9650f2e9dbe4f581: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.197:35757
I20240502 01:53:51.349550 16868 ts_tablet_manager.cc:902] T f2af8f5d5e784e1988a957666844db6b P a82548707a5844ce9650f2e9dbe4f581: Initiating tablet copy from peer b4abc88ed2e54830ae3a317629916117 (127.9.176.195:41211)
I20240502 01:53:51.355195 14976 tablet_copy_service.cc:140] P 7e74a3d5df6a483c95c9dabfc99577bc: Received BeginTabletCopySession request for tablet b38a1fb4d1694c609a932e8816fc3514 from peer a82548707a5844ce9650f2e9dbe4f581 ({username='slave'} at 127.9.176.200:56093)
I20240502 01:53:51.355592 14976 tablet_copy_service.cc:161] P 7e74a3d5df6a483c95c9dabfc99577bc: Beginning new tablet copy session on tablet b38a1fb4d1694c609a932e8816fc3514 from peer a82548707a5844ce9650f2e9dbe4f581 at {username='slave'} at 127.9.176.200:56093: session id = a82548707a5844ce9650f2e9dbe4f581-b38a1fb4d1694c609a932e8816fc3514
I20240502 01:53:51.357476 16868 tablet_copy_client.cc:308] T f2af8f5d5e784e1988a957666844db6b P a82548707a5844ce9650f2e9dbe4f581: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.195:41211
I20240502 01:53:51.359967 14976 tablet_copy_source_session.cc:205] T b38a1fb4d1694c609a932e8816fc3514 P 7e74a3d5df6a483c95c9dabfc99577bc: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:51.362301 16863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b38a1fb4d1694c609a932e8816fc3514. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:51.366995 13756 tablet_copy_service.cc:140] P b4abc88ed2e54830ae3a317629916117: Received BeginTabletCopySession request for tablet f2af8f5d5e784e1988a957666844db6b from peer a82548707a5844ce9650f2e9dbe4f581 ({username='slave'} at 127.9.176.200:50457)
I20240502 01:53:51.367228 13756 tablet_copy_service.cc:161] P b4abc88ed2e54830ae3a317629916117: Beginning new tablet copy session on tablet f2af8f5d5e784e1988a957666844db6b from peer a82548707a5844ce9650f2e9dbe4f581 at {username='slave'} at 127.9.176.200:50457: session id = a82548707a5844ce9650f2e9dbe4f581-f2af8f5d5e784e1988a957666844db6b
I20240502 01:53:51.370103 16863 tablet_copy_client.cc:783] T b38a1fb4d1694c609a932e8816fc3514 P a82548707a5844ce9650f2e9dbe4f581: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:51.370440 16863 tablet_copy_client.cc:647] T b38a1fb4d1694c609a932e8816fc3514 P a82548707a5844ce9650f2e9dbe4f581: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:51.370529 13756 tablet_copy_source_session.cc:205] T f2af8f5d5e784e1988a957666844db6b P b4abc88ed2e54830ae3a317629916117: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:51.372486 16868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2af8f5d5e784e1988a957666844db6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:51.372849 16857 log.cc:824] T bdf6095f84aa497ba5db95d55c010085 P a82548707a5844ce9650f2e9dbe4f581: Log is configured to *not* fsync() on all Append() calls
I20240502 01:53:51.375022 16863 tablet_copy_client.cc:523] T b38a1fb4d1694c609a932e8816fc3514 P a82548707a5844ce9650f2e9dbe4f581: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:51.379030 16868 tablet_copy_client.cc:783] T f2af8f5d5e784e1988a957666844db6b P a82548707a5844ce9650f2e9dbe4f581: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:51.379365 16868 tablet_copy_client.cc:647] T f2af8f5d5e784e1988a957666844db6b P a82548707a5844ce9650f2e9dbe4f581: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:51.381498 16863 tablet_bootstrap.cc:492] T b38a1fb4d1694c609a932e8816fc3514 P a82548707a5844ce9650f2e9dbe4f581: Bootstrap starting.
I20240502 01:53:51.383270 16868 tablet_copy_client.cc:523] T f2af8f5d5e784e1988a957666844db6b P a82548707a5844ce9650f2e9dbe4f581: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:51.395977 16868 tablet_bootstrap.cc:492] T f2af8f5d5e784e1988a957666844db6b P a82548707a5844ce9650f2e9dbe4f581: Bootstrap starting.
I20240502 01:53:51.411693 16875 ts_tablet_manager.cc:902] T 6682ed5cba8448c7b3cf74ab726a18e7 P a82548707a5844ce9650f2e9dbe4f581: Initiating tablet copy from peer 5359629f948641f4b56e35ad6b6a15c4 (127.9.176.193:33849)
I20240502 01:53:51.445247 16875 tablet_copy_client.cc:308] T 6682ed5cba8448c7b3cf74ab726a18e7 P a82548707a5844ce9650f2e9dbe4f581: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.193:33849
I20240502 01:53:51.446352 13491 tablet_copy_service.cc:140] P 5359629f948641f4b56e35ad6b6a15c4: Received BeginTabletCopySession request for tablet 6682ed5cba8448c7b3cf74ab726a18e7 from peer a82548707a5844ce9650f2e9dbe4f581 ({username='slave'} at 127.9.176.200:39765)
I20240502 01:53:51.446571 13491 tablet_copy_service.cc:161] P 5359629f948641f4b56e35ad6b6a15c4: Beginning new tablet copy session on tablet 6682ed5cba8448c7b3cf74ab726a18e7 from peer a82548707a5844ce9650f2e9dbe4f581 at {username='slave'} at 127.9.176.200:39765: session id = a82548707a5844ce9650f2e9dbe4f581-6682ed5cba8448c7b3cf74ab726a18e7
I20240502 01:53:51.449759 13491 tablet_copy_source_session.cc:205] T 6682ed5cba8448c7b3cf74ab726a18e7 P 5359629f948641f4b56e35ad6b6a15c4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:51.452306 16875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6682ed5cba8448c7b3cf74ab726a18e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:51.461822 16875 tablet_copy_client.cc:783] T 6682ed5cba8448c7b3cf74ab726a18e7 P a82548707a5844ce9650f2e9dbe4f581: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:51.462365 16875 tablet_copy_client.cc:647] T 6682ed5cba8448c7b3cf74ab726a18e7 P a82548707a5844ce9650f2e9dbe4f581: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:51.466099 16875 tablet_copy_client.cc:523] T 6682ed5cba8448c7b3cf74ab726a18e7 P a82548707a5844ce9650f2e9dbe4f581: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:51.484848 16875 tablet_bootstrap.cc:492] T 6682ed5cba8448c7b3cf74ab726a18e7 P a82548707a5844ce9650f2e9dbe4f581: Bootstrap starting.
I20240502 01:53:51.951500 16857 tablet_bootstrap.cc:492] T bdf6095f84aa497ba5db95d55c010085 P a82548707a5844ce9650f2e9dbe4f581: Bootstrap replayed 1/1 log segments. Stats: ops{read=312 overwritten=0 applied=312 ignored=0} inserts{seen=981 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:53:51.952280 16857 tablet_bootstrap.cc:492] T bdf6095f84aa497ba5db95d55c010085 P a82548707a5844ce9650f2e9dbe4f581: Bootstrap complete.
I20240502 01:53:51.953006 16857 ts_tablet_manager.cc:1367] T bdf6095f84aa497ba5db95d55c010085 P a82548707a5844ce9650f2e9dbe4f581: Time spent bootstrapping tablet: real 0.635s	user 0.410s	sys 0.055s
I20240502 01:53:51.962292 16857 raft_consensus.cc:348] T bdf6095f84aa497ba5db95d55c010085 P a82548707a5844ce9650f2e9dbe4f581 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 312 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } attrs { replace: true } } peers { permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 41329 } attrs { promote: false } } peers { permanent_uuid: "a82548707a5844ce9650f2e9dbe4f581" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 42345 } attrs { promote: true } }
I20240502 01:53:51.970309 16857 raft_consensus.cc:729] T bdf6095f84aa497ba5db95d55c010085 P a82548707a5844ce9650f2e9dbe4f581 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a82548707a5844ce9650f2e9dbe4f581, State: Initialized, Role: LEARNER
I20240502 01:53:51.971393 16857 consensus_queue.cc:260] T bdf6095f84aa497ba5db95d55c010085 P a82548707a5844ce9650f2e9dbe4f581 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 312, Last appended: 1.312, Last appended by leader: 312, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 312 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } attrs { replace: true } } peers { permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 41329 } attrs { promote: false } } peers { permanent_uuid: "a82548707a5844ce9650f2e9dbe4f581" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 42345 } attrs { promote: true } }
I20240502 01:53:51.988840 16857 ts_tablet_manager.cc:1398] T bdf6095f84aa497ba5db95d55c010085 P a82548707a5844ce9650f2e9dbe4f581: Time spent starting tablet: real 0.036s	user 0.011s	sys 0.003s
I20240502 01:53:51.990386 13491 tablet_copy_service.cc:342] P 5359629f948641f4b56e35ad6b6a15c4: Request end of tablet copy session a82548707a5844ce9650f2e9dbe4f581-bdf6095f84aa497ba5db95d55c010085 received from {username='slave'} at 127.9.176.200:39765
I20240502 01:53:51.990603 13491 tablet_copy_service.cc:433] P 5359629f948641f4b56e35ad6b6a15c4: ending tablet copy session a82548707a5844ce9650f2e9dbe4f581-bdf6095f84aa497ba5db95d55c010085 on tablet bdf6095f84aa497ba5db95d55c010085 with peer a82548707a5844ce9650f2e9dbe4f581
I20240502 01:53:52.010664 16863 tablet_bootstrap.cc:492] T b38a1fb4d1694c609a932e8816fc3514 P a82548707a5844ce9650f2e9dbe4f581: Bootstrap replayed 1/1 log segments. Stats: ops{read=316 overwritten=0 applied=316 ignored=0} inserts{seen=940 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:53:52.011487 16863 tablet_bootstrap.cc:492] T b38a1fb4d1694c609a932e8816fc3514 P a82548707a5844ce9650f2e9dbe4f581: Bootstrap complete.
I20240502 01:53:52.011972 16863 ts_tablet_manager.cc:1367] T b38a1fb4d1694c609a932e8816fc3514 P a82548707a5844ce9650f2e9dbe4f581: Time spent bootstrapping tablet: real 0.631s	user 0.440s	sys 0.029s
I20240502 01:53:52.013353 16863 raft_consensus.cc:348] T b38a1fb4d1694c609a932e8816fc3514 P a82548707a5844ce9650f2e9dbe4f581 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 316 OBSOLETE_local: false peers { permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false replace: true } } peers { permanent_uuid: "a82548707a5844ce9650f2e9dbe4f581" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 42345 } attrs { promote: true } }
I20240502 01:53:52.013803 16863 raft_consensus.cc:729] T b38a1fb4d1694c609a932e8816fc3514 P a82548707a5844ce9650f2e9dbe4f581 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: a82548707a5844ce9650f2e9dbe4f581, State: Initialized, Role: LEARNER
I20240502 01:53:52.014113 16863 consensus_queue.cc:260] T b38a1fb4d1694c609a932e8816fc3514 P a82548707a5844ce9650f2e9dbe4f581 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 316, Last appended: 2.316, Last appended by leader: 316, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 316 OBSOLETE_local: false peers { permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false replace: true } } peers { permanent_uuid: "a82548707a5844ce9650f2e9dbe4f581" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 42345 } attrs { promote: true } }
I20240502 01:53:52.014905 16863 ts_tablet_manager.cc:1398] T b38a1fb4d1694c609a932e8816fc3514 P a82548707a5844ce9650f2e9dbe4f581: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 01:53:52.016017 14976 tablet_copy_service.cc:342] P 7e74a3d5df6a483c95c9dabfc99577bc: Request end of tablet copy session a82548707a5844ce9650f2e9dbe4f581-b38a1fb4d1694c609a932e8816fc3514 received from {username='slave'} at 127.9.176.200:56093
I20240502 01:53:52.016228 14976 tablet_copy_service.cc:433] P 7e74a3d5df6a483c95c9dabfc99577bc: ending tablet copy session a82548707a5844ce9650f2e9dbe4f581-b38a1fb4d1694c609a932e8816fc3514 on tablet b38a1fb4d1694c609a932e8816fc3514 with peer a82548707a5844ce9650f2e9dbe4f581
I20240502 01:53:52.054054 16875 tablet_bootstrap.cc:492] T 6682ed5cba8448c7b3cf74ab726a18e7 P a82548707a5844ce9650f2e9dbe4f581: Bootstrap replayed 1/1 log segments. Stats: ops{read=304 overwritten=0 applied=304 ignored=0} inserts{seen=976 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:53:52.054519 16875 tablet_bootstrap.cc:492] T 6682ed5cba8448c7b3cf74ab726a18e7 P a82548707a5844ce9650f2e9dbe4f581: Bootstrap complete.
I20240502 01:53:52.054832 16875 ts_tablet_manager.cc:1367] T 6682ed5cba8448c7b3cf74ab726a18e7 P a82548707a5844ce9650f2e9dbe4f581: Time spent bootstrapping tablet: real 0.570s	user 0.387s	sys 0.046s
I20240502 01:53:52.055553 16875 raft_consensus.cc:348] T 6682ed5cba8448c7b3cf74ab726a18e7 P a82548707a5844ce9650f2e9dbe4f581 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 304 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } attrs { replace: true } } peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "a82548707a5844ce9650f2e9dbe4f581" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 42345 } attrs { promote: true } }
I20240502 01:53:52.055828 16875 raft_consensus.cc:729] T 6682ed5cba8448c7b3cf74ab726a18e7 P a82548707a5844ce9650f2e9dbe4f581 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a82548707a5844ce9650f2e9dbe4f581, State: Initialized, Role: LEARNER
I20240502 01:53:52.056051 16875 consensus_queue.cc:260] T 6682ed5cba8448c7b3cf74ab726a18e7 P a82548707a5844ce9650f2e9dbe4f581 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 304, Last appended: 1.304, Last appended by leader: 304, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 304 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } attrs { replace: true } } peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "a82548707a5844ce9650f2e9dbe4f581" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 42345 } attrs { promote: true } }
I20240502 01:53:52.056591 16875 ts_tablet_manager.cc:1398] T 6682ed5cba8448c7b3cf74ab726a18e7 P a82548707a5844ce9650f2e9dbe4f581: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.004s
I20240502 01:53:52.057564 13491 tablet_copy_service.cc:342] P 5359629f948641f4b56e35ad6b6a15c4: Request end of tablet copy session a82548707a5844ce9650f2e9dbe4f581-6682ed5cba8448c7b3cf74ab726a18e7 received from {username='slave'} at 127.9.176.200:39765
I20240502 01:53:52.057745 13491 tablet_copy_service.cc:433] P 5359629f948641f4b56e35ad6b6a15c4: ending tablet copy session a82548707a5844ce9650f2e9dbe4f581-6682ed5cba8448c7b3cf74ab726a18e7 on tablet 6682ed5cba8448c7b3cf74ab726a18e7 with peer a82548707a5844ce9650f2e9dbe4f581
I20240502 01:53:52.073189 16868 tablet_bootstrap.cc:492] T f2af8f5d5e784e1988a957666844db6b P a82548707a5844ce9650f2e9dbe4f581: Bootstrap replayed 1/1 log segments. Stats: ops{read=295 overwritten=0 applied=295 ignored=0} inserts{seen=923 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:53:52.073654 16868 tablet_bootstrap.cc:492] T f2af8f5d5e784e1988a957666844db6b P a82548707a5844ce9650f2e9dbe4f581: Bootstrap complete.
I20240502 01:53:52.073945 16868 ts_tablet_manager.cc:1367] T f2af8f5d5e784e1988a957666844db6b P a82548707a5844ce9650f2e9dbe4f581: Time spent bootstrapping tablet: real 0.678s	user 0.361s	sys 0.077s
I20240502 01:53:52.074615 16868 raft_consensus.cc:348] T f2af8f5d5e784e1988a957666844db6b P a82548707a5844ce9650f2e9dbe4f581 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 295 OBSOLETE_local: false peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "7505fefeaf804ed8aeb3672f386ab959" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 38645 } attrs { promote: false } } peers { permanent_uuid: "a82548707a5844ce9650f2e9dbe4f581" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 42345 } attrs { promote: true } }
I20240502 01:53:52.074836 16868 raft_consensus.cc:729] T f2af8f5d5e784e1988a957666844db6b P a82548707a5844ce9650f2e9dbe4f581 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a82548707a5844ce9650f2e9dbe4f581, State: Initialized, Role: LEARNER
I20240502 01:53:52.075027 16868 consensus_queue.cc:260] T f2af8f5d5e784e1988a957666844db6b P a82548707a5844ce9650f2e9dbe4f581 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 295, Last appended: 1.295, Last appended by leader: 295, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 295 OBSOLETE_local: false peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "7505fefeaf804ed8aeb3672f386ab959" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 38645 } attrs { promote: false } } peers { permanent_uuid: "a82548707a5844ce9650f2e9dbe4f581" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 42345 } attrs { promote: true } }
I20240502 01:53:52.075592 16868 ts_tablet_manager.cc:1398] T f2af8f5d5e784e1988a957666844db6b P a82548707a5844ce9650f2e9dbe4f581: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240502 01:53:52.076705 13756 tablet_copy_service.cc:342] P b4abc88ed2e54830ae3a317629916117: Request end of tablet copy session a82548707a5844ce9650f2e9dbe4f581-f2af8f5d5e784e1988a957666844db6b received from {username='slave'} at 127.9.176.200:50457
I20240502 01:53:52.076880 13756 tablet_copy_service.cc:433] P b4abc88ed2e54830ae3a317629916117: ending tablet copy session a82548707a5844ce9650f2e9dbe4f581-f2af8f5d5e784e1988a957666844db6b on tablet f2af8f5d5e784e1988a957666844db6b with peer a82548707a5844ce9650f2e9dbe4f581
I20240502 01:53:52.268594 16637 raft_consensus.cc:1203] T bdf6095f84aa497ba5db95d55c010085 P a82548707a5844ce9650f2e9dbe4f581 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.311->[1.312-1.312]   Dedup: 1.312->[]
I20240502 01:53:52.273118 16637 raft_consensus.cc:1203] T 6682ed5cba8448c7b3cf74ab726a18e7 P a82548707a5844ce9650f2e9dbe4f581 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.303->[1.304-1.304]   Dedup: 1.304->[]
I20240502 01:53:52.346683 16637 raft_consensus.cc:1203] T f2af8f5d5e784e1988a957666844db6b P a82548707a5844ce9650f2e9dbe4f581 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.294->[1.295-1.295]   Dedup: 1.295->[]
I20240502 01:53:52.411970 16637 raft_consensus.cc:1203] T b38a1fb4d1694c609a932e8816fc3514 P a82548707a5844ce9650f2e9dbe4f581 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.315->[2.316-2.316]   Dedup: 2.316->[]
I20240502 01:53:52.701673 16909 raft_consensus.cc:1050] T 6682ed5cba8448c7b3cf74ab726a18e7 P 5359629f948641f4b56e35ad6b6a15c4: attempting to promote NON_VOTER a82548707a5844ce9650f2e9dbe4f581 to VOTER
I20240502 01:53:52.702605 16909 consensus_queue.cc:237] T 6682ed5cba8448c7b3cf74ab726a18e7 P 5359629f948641f4b56e35ad6b6a15c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 304, Committed index: 304, Last appended: 1.304, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 305 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } attrs { replace: true } } peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "a82548707a5844ce9650f2e9dbe4f581" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 42345 } attrs { promote: false } }
I20240502 01:53:52.705171 13604 raft_consensus.cc:1261] T 6682ed5cba8448c7b3cf74ab726a18e7 P 8446b7a7ea67486288d174eea7a3b3c6 [term 1 FOLLOWER]: Refusing update from remote peer 5359629f948641f4b56e35ad6b6a15c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 304. Preceding OpId from leader: term: 1 index: 305. (index mismatch)
I20240502 01:53:52.705268 16637 raft_consensus.cc:1261] T 6682ed5cba8448c7b3cf74ab726a18e7 P a82548707a5844ce9650f2e9dbe4f581 [term 1 LEARNER]: Refusing update from remote peer 5359629f948641f4b56e35ad6b6a15c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 304. Preceding OpId from leader: term: 1 index: 305. (index mismatch)
I20240502 01:53:52.705873 16910 consensus_queue.cc:1035] T 6682ed5cba8448c7b3cf74ab726a18e7 P 5359629f948641f4b56e35ad6b6a15c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 305, Last known committed idx: 304, Time since last communication: 0.000s
I20240502 01:53:52.706302 16779 consensus_queue.cc:1035] T 6682ed5cba8448c7b3cf74ab726a18e7 P 5359629f948641f4b56e35ad6b6a15c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a82548707a5844ce9650f2e9dbe4f581" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 42345 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 305, Last known committed idx: 304, Time since last communication: 0.000s
I20240502 01:53:52.709663 16779 raft_consensus.cc:2932] T 6682ed5cba8448c7b3cf74ab726a18e7 P 5359629f948641f4b56e35ad6b6a15c4 [term 1 LEADER]: Committing config change with OpId 1.305: config changed from index 304 to 305, a82548707a5844ce9650f2e9dbe4f581 (127.9.176.200) changed from NON_VOTER to VOTER. New config: { opid_index: 305 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } attrs { replace: true } } peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "a82548707a5844ce9650f2e9dbe4f581" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 42345 } attrs { promote: false } } }
I20240502 01:53:52.710539 13604 raft_consensus.cc:2932] T 6682ed5cba8448c7b3cf74ab726a18e7 P 8446b7a7ea67486288d174eea7a3b3c6 [term 1 FOLLOWER]: Committing config change with OpId 1.305: config changed from index 304 to 305, a82548707a5844ce9650f2e9dbe4f581 (127.9.176.200) changed from NON_VOTER to VOTER. New config: { opid_index: 305 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } attrs { replace: true } } peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "a82548707a5844ce9650f2e9dbe4f581" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 42345 } attrs { promote: false } } }
I20240502 01:53:52.717471 13326 catalog_manager.cc:5484] T 6682ed5cba8448c7b3cf74ab726a18e7 P 5359629f948641f4b56e35ad6b6a15c4 reported cstate change: config changed from index 304 to 305, a82548707a5844ce9650f2e9dbe4f581 (127.9.176.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5359629f948641f4b56e35ad6b6a15c4" committed_config { opid_index: 305 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a82548707a5844ce9650f2e9dbe4f581" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 42345 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:52.719496 16637 raft_consensus.cc:2932] T 6682ed5cba8448c7b3cf74ab726a18e7 P a82548707a5844ce9650f2e9dbe4f581 [term 1 FOLLOWER]: Committing config change with OpId 1.305: config changed from index 304 to 305, a82548707a5844ce9650f2e9dbe4f581 (127.9.176.200) changed from NON_VOTER to VOTER. New config: { opid_index: 305 OBSOLETE_local: false peers { permanent_uuid: "8446b7a7ea67486288d174eea7a3b3c6" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34851 } attrs { replace: true } } peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "a82548707a5844ce9650f2e9dbe4f581" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 42345 } attrs { promote: false } } }
I20240502 01:53:52.726557 13470 consensus_queue.cc:237] T 6682ed5cba8448c7b3cf74ab726a18e7 P 5359629f948641f4b56e35ad6b6a15c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 305, Committed index: 305, Last appended: 1.305, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 306 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "a82548707a5844ce9650f2e9dbe4f581" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 42345 } attrs { promote: false } }
I20240502 01:53:52.732300 16636 raft_consensus.cc:1261] T 6682ed5cba8448c7b3cf74ab726a18e7 P a82548707a5844ce9650f2e9dbe4f581 [term 1 FOLLOWER]: Refusing update from remote peer 5359629f948641f4b56e35ad6b6a15c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 305. Preceding OpId from leader: term: 1 index: 306. (index mismatch)
I20240502 01:53:52.733482 16779 consensus_queue.cc:1035] T 6682ed5cba8448c7b3cf74ab726a18e7 P 5359629f948641f4b56e35ad6b6a15c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a82548707a5844ce9650f2e9dbe4f581" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 42345 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 306, Last known committed idx: 305, Time since last communication: 0.000s
I20240502 01:53:52.736873 16909 raft_consensus.cc:2932] T 6682ed5cba8448c7b3cf74ab726a18e7 P 5359629f948641f4b56e35ad6b6a15c4 [term 1 LEADER]: Committing config change with OpId 1.306: config changed from index 305 to 306, VOTER 8446b7a7ea67486288d174eea7a3b3c6 (127.9.176.194) evicted. New config: { opid_index: 306 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "a82548707a5844ce9650f2e9dbe4f581" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 42345 } attrs { promote: false } } }
I20240502 01:53:52.737674 16636 raft_consensus.cc:2932] T 6682ed5cba8448c7b3cf74ab726a18e7 P a82548707a5844ce9650f2e9dbe4f581 [term 1 FOLLOWER]: Committing config change with OpId 1.306: config changed from index 305 to 306, VOTER 8446b7a7ea67486288d174eea7a3b3c6 (127.9.176.194) evicted. New config: { opid_index: 306 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "a82548707a5844ce9650f2e9dbe4f581" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 42345 } attrs { promote: false } } }
I20240502 01:53:52.744207 13313 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6682ed5cba8448c7b3cf74ab726a18e7 with cas_config_opid_index 305: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:53:52.744110 13319 catalog_manager.cc:5484] T 6682ed5cba8448c7b3cf74ab726a18e7 P a82548707a5844ce9650f2e9dbe4f581 reported cstate change: config changed from index 305 to 306, VOTER 8446b7a7ea67486288d174eea7a3b3c6 (127.9.176.194) evicted. New cstate: current_term: 1 leader_uuid: "5359629f948641f4b56e35ad6b6a15c4" committed_config { opid_index: 306 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "a82548707a5844ce9650f2e9dbe4f581" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 42345 } attrs { promote: false } } }
I20240502 01:53:52.750135 13584 tablet_service.cc:1514] Processing DeleteTablet for tablet 6682ed5cba8448c7b3cf74ab726a18e7 with delete_type TABLET_DATA_TOMBSTONED (TS 8446b7a7ea67486288d174eea7a3b3c6 not found in new config with opid_index 306) from {username='slave'} at 127.0.0.1:41580
I20240502 01:53:52.752704 16779 raft_consensus.cc:1050] T bdf6095f84aa497ba5db95d55c010085 P 5359629f948641f4b56e35ad6b6a15c4: attempting to promote NON_VOTER a82548707a5844ce9650f2e9dbe4f581 to VOTER
I20240502 01:53:52.754158 16779 consensus_queue.cc:237] T bdf6095f84aa497ba5db95d55c010085 P 5359629f948641f4b56e35ad6b6a15c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 312, Committed index: 312, Last appended: 1.312, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 313 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } attrs { replace: true } } peers { permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 41329 } attrs { promote: false } } peers { permanent_uuid: "a82548707a5844ce9650f2e9dbe4f581" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 42345 } attrs { promote: false } }
I20240502 01:53:52.764436 16636 raft_consensus.cc:1261] T bdf6095f84aa497ba5db95d55c010085 P a82548707a5844ce9650f2e9dbe4f581 [term 1 LEARNER]: Refusing update from remote peer 5359629f948641f4b56e35ad6b6a15c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 312. Preceding OpId from leader: term: 1 index: 313. (index mismatch)
I20240502 01:53:52.765304 16909 consensus_queue.cc:1035] T bdf6095f84aa497ba5db95d55c010085 P 5359629f948641f4b56e35ad6b6a15c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a82548707a5844ce9650f2e9dbe4f581" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 42345 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 313, Last known committed idx: 312, Time since last communication: 0.000s
I20240502 01:53:52.767426 16987 tablet_replica.cc:330] T 6682ed5cba8448c7b3cf74ab726a18e7 P 8446b7a7ea67486288d174eea7a3b3c6: stopping tablet replica
I20240502 01:53:52.767707 16987 raft_consensus.cc:2229] T 6682ed5cba8448c7b3cf74ab726a18e7 P 8446b7a7ea67486288d174eea7a3b3c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:52.767863 14820 raft_consensus.cc:1261] T bdf6095f84aa497ba5db95d55c010085 P ce90a4bd06ba41dd8e5aa227de9780a5 [term 1 FOLLOWER]: Refusing update from remote peer 5359629f948641f4b56e35ad6b6a15c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 312. Preceding OpId from leader: term: 1 index: 313. (index mismatch)
I20240502 01:53:52.768576 16779 consensus_queue.cc:1035] T bdf6095f84aa497ba5db95d55c010085 P 5359629f948641f4b56e35ad6b6a15c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 41329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 313, Last known committed idx: 312, Time since last communication: 0.000s
I20240502 01:53:52.768247 16987 raft_consensus.cc:2258] T 6682ed5cba8448c7b3cf74ab726a18e7 P 8446b7a7ea67486288d174eea7a3b3c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:52.771479 16987 ts_tablet_manager.cc:1875] T 6682ed5cba8448c7b3cf74ab726a18e7 P 8446b7a7ea67486288d174eea7a3b3c6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:53:52.777359 16779 raft_consensus.cc:2932] T bdf6095f84aa497ba5db95d55c010085 P 5359629f948641f4b56e35ad6b6a15c4 [term 1 LEADER]: Committing config change with OpId 1.313: config changed from index 312 to 313, a82548707a5844ce9650f2e9dbe4f581 (127.9.176.200) changed from NON_VOTER to VOTER. New config: { opid_index: 313 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } attrs { replace: true } } peers { permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 41329 } attrs { promote: false } } peers { permanent_uuid: "a82548707a5844ce9650f2e9dbe4f581" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 42345 } attrs { promote: false } } }
I20240502 01:53:52.784976 13319 catalog_manager.cc:5484] T bdf6095f84aa497ba5db95d55c010085 P 5359629f948641f4b56e35ad6b6a15c4 reported cstate change: config changed from index 312 to 313, a82548707a5844ce9650f2e9dbe4f581 (127.9.176.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5359629f948641f4b56e35ad6b6a15c4" committed_config { opid_index: 313 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 41329 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a82548707a5844ce9650f2e9dbe4f581" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 42345 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:53:52.782958 16637 raft_consensus.cc:2932] T bdf6095f84aa497ba5db95d55c010085 P a82548707a5844ce9650f2e9dbe4f581 [term 1 FOLLOWER]: Committing config change with OpId 1.313: config changed from index 312 to 313, a82548707a5844ce9650f2e9dbe4f581 (127.9.176.200) changed from NON_VOTER to VOTER. New config: { opid_index: 313 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } attrs { replace: true } } peers { permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 41329 } attrs { promote: false } } peers { permanent_uuid: "a82548707a5844ce9650f2e9dbe4f581" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 42345 } attrs { promote: false } } }
I20240502 01:53:52.790079 16987 ts_tablet_manager.cc:1888] T 6682ed5cba8448c7b3cf74ab726a18e7 P 8446b7a7ea67486288d174eea7a3b3c6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.305
I20240502 01:53:52.790369 16987 log.cc:1196] T 6682ed5cba8448c7b3cf74ab726a18e7 P 8446b7a7ea67486288d174eea7a3b3c6: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/wal/wals/6682ed5cba8448c7b3cf74ab726a18e7
I20240502 01:53:52.791906 13310 catalog_manager.cc:4830] TS 8446b7a7ea67486288d174eea7a3b3c6 (127.9.176.194:34851): tablet 6682ed5cba8448c7b3cf74ab726a18e7 (table rebalance_test_table_3 [id=c824fe62874b40458b3bf6ce96e3242b]) successfully deleted
I20240502 01:53:52.793098 14821 raft_consensus.cc:2932] T bdf6095f84aa497ba5db95d55c010085 P ce90a4bd06ba41dd8e5aa227de9780a5 [term 1 FOLLOWER]: Committing config change with OpId 1.313: config changed from index 312 to 313, a82548707a5844ce9650f2e9dbe4f581 (127.9.176.200) changed from NON_VOTER to VOTER. New config: { opid_index: 313 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } attrs { replace: true } } peers { permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 41329 } attrs { promote: false } } peers { permanent_uuid: "a82548707a5844ce9650f2e9dbe4f581" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 42345 } attrs { promote: false } } }
I20240502 01:53:52.845152 13470 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "bdf6095f84aa497ba5db95d55c010085"
dest_uuid: "5359629f948641f4b56e35ad6b6a15c4"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:40668
I20240502 01:53:52.845405 13470 raft_consensus.cc:595] T bdf6095f84aa497ba5db95d55c010085 P 5359629f948641f4b56e35ad6b6a15c4 [term 1 LEADER]: Received request to transfer leadership
I20240502 01:53:52.889042 13470 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "bdf6095f84aa497ba5db95d55c010085"
dest_uuid: "5359629f948641f4b56e35ad6b6a15c4"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:40688
I20240502 01:53:52.889333 13470 raft_consensus.cc:595] T bdf6095f84aa497ba5db95d55c010085 P 5359629f948641f4b56e35ad6b6a15c4 [term 1 LEADER]: Received request to transfer leadership
I20240502 01:53:52.927516 17037 raft_consensus.cc:1050] T f2af8f5d5e784e1988a957666844db6b P b4abc88ed2e54830ae3a317629916117: attempting to promote NON_VOTER a82548707a5844ce9650f2e9dbe4f581 to VOTER
I20240502 01:53:52.928539 17037 consensus_queue.cc:237] T f2af8f5d5e784e1988a957666844db6b P b4abc88ed2e54830ae3a317629916117 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 295, Committed index: 295, Last appended: 1.295, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 296 OBSOLETE_local: false peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "7505fefeaf804ed8aeb3672f386ab959" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 38645 } attrs { promote: false } } peers { permanent_uuid: "a82548707a5844ce9650f2e9dbe4f581" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 42345 } attrs { promote: false } }
I20240502 01:53:52.931290 16637 raft_consensus.cc:1261] T f2af8f5d5e784e1988a957666844db6b P a82548707a5844ce9650f2e9dbe4f581 [term 1 LEARNER]: Refusing update from remote peer b4abc88ed2e54830ae3a317629916117: Log matching property violated. Preceding OpId in replica: term: 1 index: 295. Preceding OpId from leader: term: 1 index: 296. (index mismatch)
I20240502 01:53:52.931321 15226 raft_consensus.cc:1261] T f2af8f5d5e784e1988a957666844db6b P 7505fefeaf804ed8aeb3672f386ab959 [term 1 FOLLOWER]: Refusing update from remote peer b4abc88ed2e54830ae3a317629916117: Log matching property violated. Preceding OpId in replica: term: 1 index: 295. Preceding OpId from leader: term: 1 index: 296. (index mismatch)
I20240502 01:53:52.931988 17036 consensus_queue.cc:1035] T f2af8f5d5e784e1988a957666844db6b P b4abc88ed2e54830ae3a317629916117 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a82548707a5844ce9650f2e9dbe4f581" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 42345 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 296, Last known committed idx: 295, Time since last communication: 0.000s
I20240502 01:53:52.932397 16908 consensus_queue.cc:1035] T f2af8f5d5e784e1988a957666844db6b P b4abc88ed2e54830ae3a317629916117 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7505fefeaf804ed8aeb3672f386ab959" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 38645 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 296, Last known committed idx: 295, Time since last communication: 0.000s
I20240502 01:53:52.936100 17036 raft_consensus.cc:2932] T f2af8f5d5e784e1988a957666844db6b P b4abc88ed2e54830ae3a317629916117 [term 1 LEADER]: Committing config change with OpId 1.296: config changed from index 295 to 296, a82548707a5844ce9650f2e9dbe4f581 (127.9.176.200) changed from NON_VOTER to VOTER. New config: { opid_index: 296 OBSOLETE_local: false peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "7505fefeaf804ed8aeb3672f386ab959" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 38645 } attrs { promote: false } } peers { permanent_uuid: "a82548707a5844ce9650f2e9dbe4f581" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 42345 } attrs { promote: false } } }
I20240502 01:53:52.941082 15226 raft_consensus.cc:2932] T f2af8f5d5e784e1988a957666844db6b P 7505fefeaf804ed8aeb3672f386ab959 [term 1 FOLLOWER]: Committing config change with OpId 1.296: config changed from index 295 to 296, a82548707a5844ce9650f2e9dbe4f581 (127.9.176.200) changed from NON_VOTER to VOTER. New config: { opid_index: 296 OBSOLETE_local: false peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "7505fefeaf804ed8aeb3672f386ab959" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 38645 } attrs { promote: false } } peers { permanent_uuid: "a82548707a5844ce9650f2e9dbe4f581" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 42345 } attrs { promote: false } } }
I20240502 01:53:52.946710 13319 catalog_manager.cc:5484] T f2af8f5d5e784e1988a957666844db6b P b4abc88ed2e54830ae3a317629916117 reported cstate change: config changed from index 295 to 296, a82548707a5844ce9650f2e9dbe4f581 (127.9.176.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b4abc88ed2e54830ae3a317629916117" committed_config { opid_index: 296 OBSOLETE_local: false peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7505fefeaf804ed8aeb3672f386ab959" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 38645 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a82548707a5844ce9650f2e9dbe4f581" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 42345 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:53:52.949949 16637 raft_consensus.cc:2932] T f2af8f5d5e784e1988a957666844db6b P a82548707a5844ce9650f2e9dbe4f581 [term 1 FOLLOWER]: Committing config change with OpId 1.296: config changed from index 295 to 296, a82548707a5844ce9650f2e9dbe4f581 (127.9.176.200) changed from NON_VOTER to VOTER. New config: { opid_index: 296 OBSOLETE_local: false peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "7505fefeaf804ed8aeb3672f386ab959" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 38645 } attrs { promote: false } } peers { permanent_uuid: "a82548707a5844ce9650f2e9dbe4f581" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 42345 } attrs { promote: false } } }
I20240502 01:53:52.962175 17044 raft_consensus.cc:1050] T b38a1fb4d1694c609a932e8816fc3514 P 7e74a3d5df6a483c95c9dabfc99577bc: attempting to promote NON_VOTER a82548707a5844ce9650f2e9dbe4f581 to VOTER
I20240502 01:53:52.963419 17044 consensus_queue.cc:237] T b38a1fb4d1694c609a932e8816fc3514 P 7e74a3d5df6a483c95c9dabfc99577bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 316, Committed index: 316, Last appended: 2.316, Last appended by leader: 185, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 317 OBSOLETE_local: false peers { permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false replace: true } } peers { permanent_uuid: "a82548707a5844ce9650f2e9dbe4f581" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 42345 } attrs { promote: false } }
I20240502 01:53:52.966403 16637 raft_consensus.cc:1261] T b38a1fb4d1694c609a932e8816fc3514 P a82548707a5844ce9650f2e9dbe4f581 [term 2 LEARNER]: Refusing update from remote peer 7e74a3d5df6a483c95c9dabfc99577bc: Log matching property violated. Preceding OpId in replica: term: 2 index: 316. Preceding OpId from leader: term: 2 index: 317. (index mismatch)
I20240502 01:53:52.966712 15085 raft_consensus.cc:1261] T b38a1fb4d1694c609a932e8816fc3514 P 9e4fa889fab047baa5bd3dcd77b2ca8f [term 2 FOLLOWER]: Refusing update from remote peer 7e74a3d5df6a483c95c9dabfc99577bc: Log matching property violated. Preceding OpId in replica: term: 2 index: 316. Preceding OpId from leader: term: 2 index: 317. (index mismatch)
I20240502 01:53:52.967371 17043 consensus_queue.cc:1035] T b38a1fb4d1694c609a932e8816fc3514 P 7e74a3d5df6a483c95c9dabfc99577bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a82548707a5844ce9650f2e9dbe4f581" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 42345 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 317, Last known committed idx: 316, Time since last communication: 0.000s
I20240502 01:53:52.967866 17044 consensus_queue.cc:1035] T b38a1fb4d1694c609a932e8816fc3514 P 7e74a3d5df6a483c95c9dabfc99577bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 317, Last known committed idx: 316, Time since last communication: 0.000s
I20240502 01:53:52.971071 17044 raft_consensus.cc:2932] T b38a1fb4d1694c609a932e8816fc3514 P 7e74a3d5df6a483c95c9dabfc99577bc [term 2 LEADER]: Committing config change with OpId 2.317: config changed from index 316 to 317, a82548707a5844ce9650f2e9dbe4f581 (127.9.176.200) changed from NON_VOTER to VOTER. New config: { opid_index: 317 OBSOLETE_local: false peers { permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false replace: true } } peers { permanent_uuid: "a82548707a5844ce9650f2e9dbe4f581" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 42345 } attrs { promote: false } } }
I20240502 01:53:52.973073 15085 raft_consensus.cc:2932] T b38a1fb4d1694c609a932e8816fc3514 P 9e4fa889fab047baa5bd3dcd77b2ca8f [term 2 FOLLOWER]: Committing config change with OpId 2.317: config changed from index 316 to 317, a82548707a5844ce9650f2e9dbe4f581 (127.9.176.200) changed from NON_VOTER to VOTER. New config: { opid_index: 317 OBSOLETE_local: false peers { permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false replace: true } } peers { permanent_uuid: "a82548707a5844ce9650f2e9dbe4f581" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 42345 } attrs { promote: false } } }
I20240502 01:53:52.973768 16637 raft_consensus.cc:2932] T b38a1fb4d1694c609a932e8816fc3514 P a82548707a5844ce9650f2e9dbe4f581 [term 2 FOLLOWER]: Committing config change with OpId 2.317: config changed from index 316 to 317, a82548707a5844ce9650f2e9dbe4f581 (127.9.176.200) changed from NON_VOTER to VOTER. New config: { opid_index: 317 OBSOLETE_local: false peers { permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false replace: true } } peers { permanent_uuid: "a82548707a5844ce9650f2e9dbe4f581" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 42345 } attrs { promote: false } } }
I20240502 01:53:52.976974 13319 catalog_manager.cc:5484] T b38a1fb4d1694c609a932e8816fc3514 P 7e74a3d5df6a483c95c9dabfc99577bc reported cstate change: config changed from index 316 to 317, a82548707a5844ce9650f2e9dbe4f581 (127.9.176.200) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" committed_config { opid_index: 317 OBSOLETE_local: false peers { permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e4fa889fab047baa5bd3dcd77b2ca8f" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 38527 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a82548707a5844ce9650f2e9dbe4f581" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 42345 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:53:52.982079 14954 consensus_queue.cc:237] T b38a1fb4d1694c609a932e8816fc3514 P 7e74a3d5df6a483c95c9dabfc99577bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 317, Committed index: 317, Last appended: 2.317, Last appended by leader: 185, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 318 OBSOLETE_local: false peers { permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "a82548707a5844ce9650f2e9dbe4f581" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 42345 } attrs { promote: false } }
I20240502 01:53:52.983947 16637 raft_consensus.cc:1261] T b38a1fb4d1694c609a932e8816fc3514 P a82548707a5844ce9650f2e9dbe4f581 [term 2 FOLLOWER]: Refusing update from remote peer 7e74a3d5df6a483c95c9dabfc99577bc: Log matching property violated. Preceding OpId in replica: term: 2 index: 317. Preceding OpId from leader: term: 2 index: 318. (index mismatch)
I20240502 01:53:52.984670 17043 consensus_queue.cc:1035] T b38a1fb4d1694c609a932e8816fc3514 P 7e74a3d5df6a483c95c9dabfc99577bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a82548707a5844ce9650f2e9dbe4f581" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 42345 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 318, Last known committed idx: 317, Time since last communication: 0.000s
I20240502 01:53:52.987373 17044 raft_consensus.cc:2932] T b38a1fb4d1694c609a932e8816fc3514 P 7e74a3d5df6a483c95c9dabfc99577bc [term 2 LEADER]: Committing config change with OpId 2.318: config changed from index 317 to 318, VOTER 9e4fa889fab047baa5bd3dcd77b2ca8f (127.9.176.198) evicted. New config: { opid_index: 318 OBSOLETE_local: false peers { permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "a82548707a5844ce9650f2e9dbe4f581" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 42345 } attrs { promote: false } } }
I20240502 01:53:52.988163 16637 raft_consensus.cc:2932] T b38a1fb4d1694c609a932e8816fc3514 P a82548707a5844ce9650f2e9dbe4f581 [term 2 FOLLOWER]: Committing config change with OpId 2.318: config changed from index 317 to 318, VOTER 9e4fa889fab047baa5bd3dcd77b2ca8f (127.9.176.198) evicted. New config: { opid_index: 318 OBSOLETE_local: false peers { permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "a82548707a5844ce9650f2e9dbe4f581" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 42345 } attrs { promote: false } } }
I20240502 01:53:52.990693 13310 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b38a1fb4d1694c609a932e8816fc3514 with cas_config_opid_index 317: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:53:52.992250 13326 catalog_manager.cc:5484] T b38a1fb4d1694c609a932e8816fc3514 P 7e74a3d5df6a483c95c9dabfc99577bc reported cstate change: config changed from index 317 to 318, VOTER 9e4fa889fab047baa5bd3dcd77b2ca8f (127.9.176.198) evicted. New cstate: current_term: 2 leader_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" committed_config { opid_index: 318 OBSOLETE_local: false peers { permanent_uuid: "7e74a3d5df6a483c95c9dabfc99577bc" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35757 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a82548707a5844ce9650f2e9dbe4f581" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 42345 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:53:52.996151 15067 tablet_service.cc:1514] Processing DeleteTablet for tablet b38a1fb4d1694c609a932e8816fc3514 with delete_type TABLET_DATA_TOMBSTONED (TS 9e4fa889fab047baa5bd3dcd77b2ca8f not found in new config with opid_index 318) from {username='slave'} at 127.0.0.1:55864
I20240502 01:53:52.997036 17048 tablet_replica.cc:330] T b38a1fb4d1694c609a932e8816fc3514 P 9e4fa889fab047baa5bd3dcd77b2ca8f: stopping tablet replica
I20240502 01:53:52.997256 17048 raft_consensus.cc:2229] T b38a1fb4d1694c609a932e8816fc3514 P 9e4fa889fab047baa5bd3dcd77b2ca8f [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:52.997642 17048 raft_consensus.cc:2258] T b38a1fb4d1694c609a932e8816fc3514 P 9e4fa889fab047baa5bd3dcd77b2ca8f [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:52.999294 17048 ts_tablet_manager.cc:1875] T b38a1fb4d1694c609a932e8816fc3514 P 9e4fa889fab047baa5bd3dcd77b2ca8f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:53:53.004060 17048 ts_tablet_manager.cc:1888] T b38a1fb4d1694c609a932e8816fc3514 P 9e4fa889fab047baa5bd3dcd77b2ca8f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.317
I20240502 01:53:53.004357 17048 log.cc:1196] T b38a1fb4d1694c609a932e8816fc3514 P 9e4fa889fab047baa5bd3dcd77b2ca8f: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/wal/wals/b38a1fb4d1694c609a932e8816fc3514
I20240502 01:53:53.005266 13313 catalog_manager.cc:4830] TS 9e4fa889fab047baa5bd3dcd77b2ca8f (127.9.176.198:38527): tablet b38a1fb4d1694c609a932e8816fc3514 (table rebalance_test_table_1 [id=8ebdfe08a0b244819813a774f9d721fc]) successfully deleted
I20240502 01:53:53.121759 13732 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f2af8f5d5e784e1988a957666844db6b"
dest_uuid: "b4abc88ed2e54830ae3a317629916117"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:43958
I20240502 01:53:53.122059 13732 raft_consensus.cc:595] T f2af8f5d5e784e1988a957666844db6b P b4abc88ed2e54830ae3a317629916117 [term 1 LEADER]: Received request to transfer leadership
I20240502 01:53:53.148046 13470 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "bdf6095f84aa497ba5db95d55c010085"
dest_uuid: "5359629f948641f4b56e35ad6b6a15c4"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:40716
I20240502 01:53:53.148353 13470 raft_consensus.cc:595] T bdf6095f84aa497ba5db95d55c010085 P 5359629f948641f4b56e35ad6b6a15c4 [term 1 LEADER]: Received request to transfer leadership
I20240502 01:53:53.171252 13732 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f2af8f5d5e784e1988a957666844db6b"
dest_uuid: "b4abc88ed2e54830ae3a317629916117"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:43976
I20240502 01:53:53.171624 13732 raft_consensus.cc:595] T f2af8f5d5e784e1988a957666844db6b P b4abc88ed2e54830ae3a317629916117 [term 1 LEADER]: Received request to transfer leadership
I20240502 01:53:53.193187 13470 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "bdf6095f84aa497ba5db95d55c010085"
dest_uuid: "5359629f948641f4b56e35ad6b6a15c4"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:40738
I20240502 01:53:53.193423 13470 raft_consensus.cc:595] T bdf6095f84aa497ba5db95d55c010085 P 5359629f948641f4b56e35ad6b6a15c4 [term 1 LEADER]: Received request to transfer leadership
I20240502 01:53:53.211151 16910 raft_consensus.cc:979] T bdf6095f84aa497ba5db95d55c010085 P 5359629f948641f4b56e35ad6b6a15c4: : Instructing follower a82548707a5844ce9650f2e9dbe4f581 to start an election
I20240502 01:53:53.211344 16910 raft_consensus.cc:1067] T bdf6095f84aa497ba5db95d55c010085 P 5359629f948641f4b56e35ad6b6a15c4 [term 1 LEADER]: Signalling peer a82548707a5844ce9650f2e9dbe4f581 to start an election
I20240502 01:53:53.212222 16637 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "bdf6095f84aa497ba5db95d55c010085"
dest_uuid: "a82548707a5844ce9650f2e9dbe4f581"
 from {username='slave'} at 127.9.176.193:47109
I20240502 01:53:53.212555 16637 raft_consensus.cc:482] T bdf6095f84aa497ba5db95d55c010085 P a82548707a5844ce9650f2e9dbe4f581 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:53:53.212744 16637 raft_consensus.cc:3037] T bdf6095f84aa497ba5db95d55c010085 P a82548707a5844ce9650f2e9dbe4f581 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:53:53.214840 16637 raft_consensus.cc:504] T bdf6095f84aa497ba5db95d55c010085 P a82548707a5844ce9650f2e9dbe4f581 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 313 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } attrs { replace: true } } peers { permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 41329 } attrs { promote: false } } peers { permanent_uuid: "a82548707a5844ce9650f2e9dbe4f581" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 42345 } attrs { promote: false } }
I20240502 01:53:53.216631 16637 leader_election.cc:288] T bdf6095f84aa497ba5db95d55c010085 P a82548707a5844ce9650f2e9dbe4f581 [CANDIDATE]: Term 2 election: Requested vote from peers 5359629f948641f4b56e35ad6b6a15c4 (127.9.176.193:33849), ce90a4bd06ba41dd8e5aa227de9780a5 (127.9.176.196:41329)
I20240502 01:53:53.217227 13470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdf6095f84aa497ba5db95d55c010085" candidate_uuid: "a82548707a5844ce9650f2e9dbe4f581" candidate_term: 2 candidate_status { last_received { term: 1 index: 313 } } ignore_live_leader: true dest_uuid: "5359629f948641f4b56e35ad6b6a15c4"
I20240502 01:53:53.217499 13470 raft_consensus.cc:3032] T bdf6095f84aa497ba5db95d55c010085 P 5359629f948641f4b56e35ad6b6a15c4 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:53:53.217607 13470 raft_consensus.cc:729] T bdf6095f84aa497ba5db95d55c010085 P 5359629f948641f4b56e35ad6b6a15c4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5359629f948641f4b56e35ad6b6a15c4, State: Running, Role: LEADER
I20240502 01:53:53.217914 13470 consensus_queue.cc:260] T bdf6095f84aa497ba5db95d55c010085 P 5359629f948641f4b56e35ad6b6a15c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 313, Committed index: 313, Last appended: 1.313, Last appended by leader: 313, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 313 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } attrs { replace: true } } peers { permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 41329 } attrs { promote: false } } peers { permanent_uuid: "a82548707a5844ce9650f2e9dbe4f581" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 42345 } attrs { promote: false } }
I20240502 01:53:53.218245 13470 raft_consensus.cc:3037] T bdf6095f84aa497ba5db95d55c010085 P 5359629f948641f4b56e35ad6b6a15c4 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:53:53.220749 13470 raft_consensus.cc:2454] T bdf6095f84aa497ba5db95d55c010085 P 5359629f948641f4b56e35ad6b6a15c4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a82548707a5844ce9650f2e9dbe4f581 in term 2.
I20240502 01:53:53.221426 16572 leader_election.cc:302] T bdf6095f84aa497ba5db95d55c010085 P a82548707a5844ce9650f2e9dbe4f581 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5359629f948641f4b56e35ad6b6a15c4, a82548707a5844ce9650f2e9dbe4f581; no voters: 
I20240502 01:53:53.221902 16923 raft_consensus.cc:2781] T bdf6095f84aa497ba5db95d55c010085 P a82548707a5844ce9650f2e9dbe4f581 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:53:53.222414 16923 raft_consensus.cc:686] T bdf6095f84aa497ba5db95d55c010085 P a82548707a5844ce9650f2e9dbe4f581 [term 2 LEADER]: Becoming Leader. State: Replica: a82548707a5844ce9650f2e9dbe4f581, State: Running, Role: LEADER
I20240502 01:53:53.223042 16923 consensus_queue.cc:237] T bdf6095f84aa497ba5db95d55c010085 P a82548707a5844ce9650f2e9dbe4f581 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 313, Committed index: 313, Last appended: 1.313, Last appended by leader: 313, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 313 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } attrs { replace: true } } peers { permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 41329 } attrs { promote: false } } peers { permanent_uuid: "a82548707a5844ce9650f2e9dbe4f581" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 42345 } attrs { promote: false } }
I20240502 01:53:53.227226 14821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdf6095f84aa497ba5db95d55c010085" candidate_uuid: "a82548707a5844ce9650f2e9dbe4f581" candidate_term: 2 candidate_status { last_received { term: 1 index: 313 } } ignore_live_leader: true dest_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5"
I20240502 01:53:53.227599 14821 raft_consensus.cc:3037] T bdf6095f84aa497ba5db95d55c010085 P ce90a4bd06ba41dd8e5aa227de9780a5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:53:53.227275 13317 catalog_manager.cc:5484] T bdf6095f84aa497ba5db95d55c010085 P a82548707a5844ce9650f2e9dbe4f581 reported cstate change: term changed from 1 to 2, leader changed from 5359629f948641f4b56e35ad6b6a15c4 (127.9.176.193) to a82548707a5844ce9650f2e9dbe4f581 (127.9.176.200). New cstate: current_term: 2 leader_uuid: "a82548707a5844ce9650f2e9dbe4f581" committed_config { opid_index: 313 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 41329 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a82548707a5844ce9650f2e9dbe4f581" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 42345 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:53:53.230412 14821 raft_consensus.cc:2454] T bdf6095f84aa497ba5db95d55c010085 P ce90a4bd06ba41dd8e5aa227de9780a5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a82548707a5844ce9650f2e9dbe4f581 in term 2.
I20240502 01:53:53.370674 17036 raft_consensus.cc:979] T f2af8f5d5e784e1988a957666844db6b P b4abc88ed2e54830ae3a317629916117: : Instructing follower 7505fefeaf804ed8aeb3672f386ab959 to start an election
I20240502 01:53:53.370903 16908 raft_consensus.cc:1067] T f2af8f5d5e784e1988a957666844db6b P b4abc88ed2e54830ae3a317629916117 [term 1 LEADER]: Signalling peer 7505fefeaf804ed8aeb3672f386ab959 to start an election
I20240502 01:53:53.372049 15226 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f2af8f5d5e784e1988a957666844db6b"
dest_uuid: "7505fefeaf804ed8aeb3672f386ab959"
 from {username='slave'} at 127.9.176.195:34201
I20240502 01:53:53.372364 15226 raft_consensus.cc:482] T f2af8f5d5e784e1988a957666844db6b P 7505fefeaf804ed8aeb3672f386ab959 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:53:53.372524 15226 raft_consensus.cc:3037] T f2af8f5d5e784e1988a957666844db6b P 7505fefeaf804ed8aeb3672f386ab959 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:53:53.374784 15226 raft_consensus.cc:504] T f2af8f5d5e784e1988a957666844db6b P 7505fefeaf804ed8aeb3672f386ab959 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 296 OBSOLETE_local: false peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "7505fefeaf804ed8aeb3672f386ab959" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 38645 } attrs { promote: false } } peers { permanent_uuid: "a82548707a5844ce9650f2e9dbe4f581" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 42345 } attrs { promote: false } }
I20240502 01:53:53.376453 15226 leader_election.cc:288] T f2af8f5d5e784e1988a957666844db6b P 7505fefeaf804ed8aeb3672f386ab959 [CANDIDATE]: Term 2 election: Requested vote from peers b4abc88ed2e54830ae3a317629916117 (127.9.176.195:41211), a82548707a5844ce9650f2e9dbe4f581 (127.9.176.200:42345)
I20240502 01:53:53.377189 13732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2af8f5d5e784e1988a957666844db6b" candidate_uuid: "7505fefeaf804ed8aeb3672f386ab959" candidate_term: 2 candidate_status { last_received { term: 1 index: 296 } } ignore_live_leader: true dest_uuid: "b4abc88ed2e54830ae3a317629916117"
I20240502 01:53:53.377475 13732 raft_consensus.cc:3032] T f2af8f5d5e784e1988a957666844db6b P b4abc88ed2e54830ae3a317629916117 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:53:53.377609 13732 raft_consensus.cc:729] T f2af8f5d5e784e1988a957666844db6b P b4abc88ed2e54830ae3a317629916117 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b4abc88ed2e54830ae3a317629916117, State: Running, Role: LEADER
I20240502 01:53:53.378576 13732 consensus_queue.cc:260] T f2af8f5d5e784e1988a957666844db6b P b4abc88ed2e54830ae3a317629916117 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 296, Committed index: 296, Last appended: 1.296, Last appended by leader: 296, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 296 OBSOLETE_local: false peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "7505fefeaf804ed8aeb3672f386ab959" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 38645 } attrs { promote: false } } peers { permanent_uuid: "a82548707a5844ce9650f2e9dbe4f581" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 42345 } attrs { promote: false } }
I20240502 01:53:53.379143 13732 raft_consensus.cc:3037] T f2af8f5d5e784e1988a957666844db6b P b4abc88ed2e54830ae3a317629916117 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:53:53.381527 13732 raft_consensus.cc:2454] T f2af8f5d5e784e1988a957666844db6b P b4abc88ed2e54830ae3a317629916117 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7505fefeaf804ed8aeb3672f386ab959 in term 2.
I20240502 01:53:53.382375 15158 leader_election.cc:302] T f2af8f5d5e784e1988a957666844db6b P 7505fefeaf804ed8aeb3672f386ab959 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7505fefeaf804ed8aeb3672f386ab959, b4abc88ed2e54830ae3a317629916117; no voters: 
I20240502 01:53:53.382939 17042 raft_consensus.cc:2781] T f2af8f5d5e784e1988a957666844db6b P 7505fefeaf804ed8aeb3672f386ab959 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:53:53.383634 17042 raft_consensus.cc:686] T f2af8f5d5e784e1988a957666844db6b P 7505fefeaf804ed8aeb3672f386ab959 [term 2 LEADER]: Becoming Leader. State: Replica: 7505fefeaf804ed8aeb3672f386ab959, State: Running, Role: LEADER
I20240502 01:53:53.384302 17042 consensus_queue.cc:237] T f2af8f5d5e784e1988a957666844db6b P 7505fefeaf804ed8aeb3672f386ab959 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 296, Committed index: 296, Last appended: 1.296, Last appended by leader: 296, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 296 OBSOLETE_local: false peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } attrs { replace: true } } peers { permanent_uuid: "7505fefeaf804ed8aeb3672f386ab959" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 38645 } attrs { promote: false } } peers { permanent_uuid: "a82548707a5844ce9650f2e9dbe4f581" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 42345 } attrs { promote: false } }
I20240502 01:53:53.384841 16637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2af8f5d5e784e1988a957666844db6b" candidate_uuid: "7505fefeaf804ed8aeb3672f386ab959" candidate_term: 2 candidate_status { last_received { term: 1 index: 296 } } ignore_live_leader: true dest_uuid: "a82548707a5844ce9650f2e9dbe4f581"
I20240502 01:53:53.385147 16637 raft_consensus.cc:3037] T f2af8f5d5e784e1988a957666844db6b P a82548707a5844ce9650f2e9dbe4f581 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:53:53.387523 16637 raft_consensus.cc:2454] T f2af8f5d5e784e1988a957666844db6b P a82548707a5844ce9650f2e9dbe4f581 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7505fefeaf804ed8aeb3672f386ab959 in term 2.
I20240502 01:53:53.389006 13317 catalog_manager.cc:5484] T f2af8f5d5e784e1988a957666844db6b P 7505fefeaf804ed8aeb3672f386ab959 reported cstate change: term changed from 1 to 2, leader changed from b4abc88ed2e54830ae3a317629916117 (127.9.176.195) to 7505fefeaf804ed8aeb3672f386ab959 (127.9.176.199). New cstate: current_term: 2 leader_uuid: "7505fefeaf804ed8aeb3672f386ab959" committed_config { opid_index: 296 OBSOLETE_local: false peers { permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7505fefeaf804ed8aeb3672f386ab959" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 38645 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a82548707a5844ce9650f2e9dbe4f581" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 42345 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:53.647629 14821 raft_consensus.cc:1261] T bdf6095f84aa497ba5db95d55c010085 P ce90a4bd06ba41dd8e5aa227de9780a5 [term 2 FOLLOWER]: Refusing update from remote peer a82548707a5844ce9650f2e9dbe4f581: Log matching property violated. Preceding OpId in replica: term: 1 index: 313. Preceding OpId from leader: term: 2 index: 314. (index mismatch)
I20240502 01:53:53.648356 16923 consensus_queue.cc:1035] T bdf6095f84aa497ba5db95d55c010085 P a82548707a5844ce9650f2e9dbe4f581 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 41329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 314, Last known committed idx: 313, Time since last communication: 0.000s
I20240502 01:53:53.664085 13470 raft_consensus.cc:1261] T bdf6095f84aa497ba5db95d55c010085 P 5359629f948641f4b56e35ad6b6a15c4 [term 2 FOLLOWER]: Refusing update from remote peer a82548707a5844ce9650f2e9dbe4f581: Log matching property violated. Preceding OpId in replica: term: 1 index: 313. Preceding OpId from leader: term: 2 index: 314. (index mismatch)
I20240502 01:53:53.664927 16923 consensus_queue.cc:1035] T bdf6095f84aa497ba5db95d55c010085 P a82548707a5844ce9650f2e9dbe4f581 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 314, Last known committed idx: 313, Time since last communication: 0.000s
I20240502 01:53:53.676604 16637 consensus_queue.cc:237] T bdf6095f84aa497ba5db95d55c010085 P a82548707a5844ce9650f2e9dbe4f581 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 314, Committed index: 314, Last appended: 2.314, Last appended by leader: 313, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 315 OBSOLETE_local: false peers { permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 41329 } attrs { promote: false } } peers { permanent_uuid: "a82548707a5844ce9650f2e9dbe4f581" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 42345 } attrs { promote: false } }
I20240502 01:53:53.678697 14821 raft_consensus.cc:1261] T bdf6095f84aa497ba5db95d55c010085 P ce90a4bd06ba41dd8e5aa227de9780a5 [term 2 FOLLOWER]: Refusing update from remote peer a82548707a5844ce9650f2e9dbe4f581: Log matching property violated. Preceding OpId in replica: term: 2 index: 314. Preceding OpId from leader: term: 2 index: 315. (index mismatch)
I20240502 01:53:53.679399 16923 consensus_queue.cc:1035] T bdf6095f84aa497ba5db95d55c010085 P a82548707a5844ce9650f2e9dbe4f581 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 41329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 315, Last known committed idx: 314, Time since last communication: 0.000s
I20240502 01:53:53.682509 17097 raft_consensus.cc:2932] T bdf6095f84aa497ba5db95d55c010085 P a82548707a5844ce9650f2e9dbe4f581 [term 2 LEADER]: Committing config change with OpId 2.315: config changed from index 313 to 315, VOTER 5359629f948641f4b56e35ad6b6a15c4 (127.9.176.193) evicted. New config: { opid_index: 315 OBSOLETE_local: false peers { permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 41329 } attrs { promote: false } } peers { permanent_uuid: "a82548707a5844ce9650f2e9dbe4f581" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 42345 } attrs { promote: false } } }
I20240502 01:53:53.683504 14821 raft_consensus.cc:2932] T bdf6095f84aa497ba5db95d55c010085 P ce90a4bd06ba41dd8e5aa227de9780a5 [term 2 FOLLOWER]: Committing config change with OpId 2.315: config changed from index 313 to 315, VOTER 5359629f948641f4b56e35ad6b6a15c4 (127.9.176.193) evicted. New config: { opid_index: 315 OBSOLETE_local: false peers { permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 41329 } attrs { promote: false } } peers { permanent_uuid: "a82548707a5844ce9650f2e9dbe4f581" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 42345 } attrs { promote: false } } }
I20240502 01:53:53.686426 13313 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet bdf6095f84aa497ba5db95d55c010085 with cas_config_opid_index 313: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:53:53.688320 13317 catalog_manager.cc:5484] T bdf6095f84aa497ba5db95d55c010085 P a82548707a5844ce9650f2e9dbe4f581 reported cstate change: config changed from index 313 to 315, VOTER 5359629f948641f4b56e35ad6b6a15c4 (127.9.176.193) evicted. New cstate: current_term: 2 leader_uuid: "a82548707a5844ce9650f2e9dbe4f581" committed_config { opid_index: 315 OBSOLETE_local: false peers { permanent_uuid: "ce90a4bd06ba41dd8e5aa227de9780a5" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 41329 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a82548707a5844ce9650f2e9dbe4f581" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 42345 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:53:53.694276 13451 tablet_service.cc:1514] Processing DeleteTablet for tablet bdf6095f84aa497ba5db95d55c010085 with delete_type TABLET_DATA_TOMBSTONED (TS 5359629f948641f4b56e35ad6b6a15c4 not found in new config with opid_index 315) from {username='slave'} at 127.0.0.1:38182
I20240502 01:53:53.694922 17128 tablet_replica.cc:330] T bdf6095f84aa497ba5db95d55c010085 P 5359629f948641f4b56e35ad6b6a15c4: stopping tablet replica
I20240502 01:53:53.695144 17128 raft_consensus.cc:2229] T bdf6095f84aa497ba5db95d55c010085 P 5359629f948641f4b56e35ad6b6a15c4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:53.695358 17128 raft_consensus.cc:2258] T bdf6095f84aa497ba5db95d55c010085 P 5359629f948641f4b56e35ad6b6a15c4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:53.696985 17128 ts_tablet_manager.cc:1875] T bdf6095f84aa497ba5db95d55c010085 P 5359629f948641f4b56e35ad6b6a15c4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:53:53.701826 17128 ts_tablet_manager.cc:1888] T bdf6095f84aa497ba5db95d55c010085 P 5359629f948641f4b56e35ad6b6a15c4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.314
I20240502 01:53:53.702001 17128 log.cc:1196] T bdf6095f84aa497ba5db95d55c010085 P 5359629f948641f4b56e35ad6b6a15c4: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/wal/wals/bdf6095f84aa497ba5db95d55c010085
I20240502 01:53:53.702915 13313 catalog_manager.cc:4830] TS 5359629f948641f4b56e35ad6b6a15c4 (127.9.176.193:33849): tablet bdf6095f84aa497ba5db95d55c010085 (table rebalance_test_table_2 [id=c4a6875e444a446dbf9576f2a5d024c0]) successfully deleted
I20240502 01:53:53.907177 13732 raft_consensus.cc:1261] T f2af8f5d5e784e1988a957666844db6b P b4abc88ed2e54830ae3a317629916117 [term 2 FOLLOWER]: Refusing update from remote peer 7505fefeaf804ed8aeb3672f386ab959: Log matching property violated. Preceding OpId in replica: term: 1 index: 296. Preceding OpId from leader: term: 2 index: 297. (index mismatch)
I20240502 01:53:53.908118 17145 consensus_queue.cc:1035] T f2af8f5d5e784e1988a957666844db6b P 7505fefeaf804ed8aeb3672f386ab959 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4abc88ed2e54830ae3a317629916117" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 41211 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 297, Last known committed idx: 296, Time since last communication: 0.000s
I20240502 01:53:53.913190 16637 raft_consensus.cc:1261] T f2af8f5d5e784e1988a957666844db6b P a82548707a5844ce9650f2e9dbe4f581 [term 2 FOLLOWER]: Refusing update from remote peer 7505fefeaf804ed8aeb3672f386ab959: Log matching property violated. Preceding OpId in replica: term: 1 index: 296. Preceding OpId from leader: term: 2 index: 297. (index mismatch)
I20240502 01:53:53.914494 17148 consensus_queue.cc:1035] T f2af8f5d5e784e1988a957666844db6b P 7505fefeaf804ed8aeb3672f386ab959 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a82548707a5844ce9650f2e9dbe4f581" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 42345 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 297, Last known committed idx: 296, Time since last communication: 0.000s
I20240502 01:53:54.405815 15226 consensus_queue.cc:237] T f2af8f5d5e784e1988a957666844db6b P 7505fefeaf804ed8aeb3672f386ab959 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 297, Committed index: 297, Last appended: 2.297, Last appended by leader: 296, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 298 OBSOLETE_local: false peers { permanent_uuid: "7505fefeaf804ed8aeb3672f386ab959" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 38645 } attrs { promote: false } } peers { permanent_uuid: "a82548707a5844ce9650f2e9dbe4f581" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 42345 } attrs { promote: false } }
I20240502 01:53:54.407724 16637 raft_consensus.cc:1261] T f2af8f5d5e784e1988a957666844db6b P a82548707a5844ce9650f2e9dbe4f581 [term 2 FOLLOWER]: Refusing update from remote peer 7505fefeaf804ed8aeb3672f386ab959: Log matching property violated. Preceding OpId in replica: term: 2 index: 297. Preceding OpId from leader: term: 2 index: 298. (index mismatch)
I20240502 01:53:54.408313 17146 consensus_queue.cc:1035] T f2af8f5d5e784e1988a957666844db6b P 7505fefeaf804ed8aeb3672f386ab959 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a82548707a5844ce9650f2e9dbe4f581" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 42345 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 298, Last known committed idx: 297, Time since last communication: 0.000s
I20240502 01:53:54.410714 17148 raft_consensus.cc:2932] T f2af8f5d5e784e1988a957666844db6b P 7505fefeaf804ed8aeb3672f386ab959 [term 2 LEADER]: Committing config change with OpId 2.298: config changed from index 296 to 298, VOTER b4abc88ed2e54830ae3a317629916117 (127.9.176.195) evicted. New config: { opid_index: 298 OBSOLETE_local: false peers { permanent_uuid: "7505fefeaf804ed8aeb3672f386ab959" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 38645 } attrs { promote: false } } peers { permanent_uuid: "a82548707a5844ce9650f2e9dbe4f581" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 42345 } attrs { promote: false } } }
I20240502 01:53:54.411410 16637 raft_consensus.cc:2932] T f2af8f5d5e784e1988a957666844db6b P a82548707a5844ce9650f2e9dbe4f581 [term 2 FOLLOWER]: Committing config change with OpId 2.298: config changed from index 296 to 298, VOTER b4abc88ed2e54830ae3a317629916117 (127.9.176.195) evicted. New config: { opid_index: 298 OBSOLETE_local: false peers { permanent_uuid: "7505fefeaf804ed8aeb3672f386ab959" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 38645 } attrs { promote: false } } peers { permanent_uuid: "a82548707a5844ce9650f2e9dbe4f581" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 42345 } attrs { promote: false } } }
I20240502 01:53:54.415452 13313 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f2af8f5d5e784e1988a957666844db6b with cas_config_opid_index 296: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:53:54.416179 13317 catalog_manager.cc:5484] T f2af8f5d5e784e1988a957666844db6b P a82548707a5844ce9650f2e9dbe4f581 reported cstate change: config changed from index 296 to 298, VOTER b4abc88ed2e54830ae3a317629916117 (127.9.176.195) evicted. New cstate: current_term: 2 leader_uuid: "7505fefeaf804ed8aeb3672f386ab959" committed_config { opid_index: 298 OBSOLETE_local: false peers { permanent_uuid: "7505fefeaf804ed8aeb3672f386ab959" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 38645 } attrs { promote: false } } peers { permanent_uuid: "a82548707a5844ce9650f2e9dbe4f581" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 42345 } attrs { promote: false } } }
I20240502 01:53:54.420542 13715 tablet_service.cc:1514] Processing DeleteTablet for tablet f2af8f5d5e784e1988a957666844db6b with delete_type TABLET_DATA_TOMBSTONED (TS b4abc88ed2e54830ae3a317629916117 not found in new config with opid_index 298) from {username='slave'} at 127.0.0.1:44986
I20240502 01:53:54.421257 17165 tablet_replica.cc:330] T f2af8f5d5e784e1988a957666844db6b P b4abc88ed2e54830ae3a317629916117: stopping tablet replica
I20240502 01:53:54.421491 17165 raft_consensus.cc:2229] T f2af8f5d5e784e1988a957666844db6b P b4abc88ed2e54830ae3a317629916117 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:54.421736 17165 raft_consensus.cc:2258] T f2af8f5d5e784e1988a957666844db6b P b4abc88ed2e54830ae3a317629916117 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:54.424340 17165 ts_tablet_manager.cc:1875] T f2af8f5d5e784e1988a957666844db6b P b4abc88ed2e54830ae3a317629916117: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:53:54.428969 17165 ts_tablet_manager.cc:1888] T f2af8f5d5e784e1988a957666844db6b P b4abc88ed2e54830ae3a317629916117: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.297
I20240502 01:53:54.429101 17165 log.cc:1196] T f2af8f5d5e784e1988a957666844db6b P b4abc88ed2e54830ae3a317629916117: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/wal/wals/f2af8f5d5e784e1988a957666844db6b
I20240502 01:53:54.429972 13312 catalog_manager.cc:4830] TS b4abc88ed2e54830ae3a317629916117 (127.9.176.195:41211): tablet f2af8f5d5e784e1988a957666844db6b (table rebalance_test_table_4 [id=7c97435ee22d42aaa3452b6e2abdaf75]) successfully deleted
I20240502 01:53:54.648044 13715 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:53:54.648157 13451 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:53:54.654186 15208 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:54.668239 14934 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:54.675009 13584 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:53:54.677284 15067 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:54.694031 16617 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:54.722457 14799 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:54.935151 13451 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:53:54.950789 15208 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:54.966595 14934 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:55.006551 16617 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:55.009675 13584 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:53:55.012197 14799 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:55.044862 13715 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:53:55.048192 15067 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:55.402464 14934 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:55.404887 13715 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:53:55.417825 15208 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:55.420766 13451 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:53:55.431492 15067 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:55.432564 13584 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:53:55.440174 16617 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:55.476238 14799 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
               UUID               |       Address       | Status
----------------------------------+---------------------+---------
 fad8aa713de7433db0f21791b371a3e4 | 127.9.176.254:44447 | HEALTHY

Unusual flags for Master:
                   Flag                   |                                                                                Value                                                                                |         Tags          |         Master
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
 allow_unsafe_replication_factor          | true                                                                                                                                                                | runtime,unsafe        | all 1 server(s) checked
 ipki_ca_key_size                         | 768                                                                                                                                                                 | experimental          | all 1 server(s) checked
 ipki_server_key_size                     | 768                                                                                                                                                                 | experimental          | all 1 server(s) checked
 never_fsync                              | true                                                                                                                                                                | unsafe,advanced       | all 1 server(s) checked
 openssl_security_level_override          | 0                                                                                                                                                                   | unsafe,hidden         | all 1 server(s) checked
 raft_prepare_replacement_before_eviction | true                                                                                                                                                                | experimental,advanced | all 1 server(s) checked
 rpc_reuseport                            | true                                                                                                                                                                | experimental          | all 1 server(s) checked
 rpc_server_allow_ephemeral_ports         | true                                                                                                                                                                | unsafe                | all 1 server(s) checked
 server_dump_info_format                  | pb                                                                                                                                                                  | hidden                | all 1 server(s) checked
 server_dump_info_path                    | /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden                | all 1 server(s) checked
 tsk_num_rsa_bits                         | 512                                                                                                                                                                 | experimental          | all 1 server(s) checked

Flags of checked categories for Master:
        Flag         |        Value        |         Master
---------------------+---------------------+-------------------------
 builtin_ntp_servers | 127.9.176.212:33337 | all 1 server(s) checked
 time_source         | builtin             | all 1 server(s) checked

Tablet Server Summary
               UUID               |       Address       | Status  | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
 5359629f948641f4b56e35ad6b6a15c4 | 127.9.176.193:33849 | HEALTHY | <none>   |       4        |       0
 7505fefeaf804ed8aeb3672f386ab959 | 127.9.176.199:38645 | HEALTHY | <none>   |       1        |       0
 7e74a3d5df6a483c95c9dabfc99577bc | 127.9.176.197:35757 | HEALTHY | <none>   |       1        |       0
 8446b7a7ea67486288d174eea7a3b3c6 | 127.9.176.194:34851 | HEALTHY | <none>   |       3        |       0
 9e4fa889fab047baa5bd3dcd77b2ca8f | 127.9.176.198:38527 | HEALTHY | <none>   |       1        |       0
 a82548707a5844ce9650f2e9dbe4f581 | 127.9.176.200:42345 | HEALTHY | <none>   |       1        |       0
 b4abc88ed2e54830ae3a317629916117 | 127.9.176.195:41211 | HEALTHY | <none>   |       4        |       0
 ce90a4bd06ba41dd8e5aa227de9780a5 | 127.9.176.196:41329 | HEALTHY | <none>   |       1        |       0

Tablet Server Location Summary
 Location |  Count
----------+---------
 <none>   |       8

Unusual flags for Tablet Server:
                   Flag                   |                                                                              Value                                                                              |         Tags          |      Tablet Server
------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
 ipki_server_key_size                     | 768                                                                                                                                                             | experimental          | all 8 server(s) checked
 local_ip_for_outbound_sockets            | 127.9.176.193                                                                                                                                                   | experimental          | 127.9.176.193:33849
 local_ip_for_outbound_sockets            | 127.9.176.194                                                                                                                                                   | experimental          | 127.9.176.194:34851
 local_ip_for_outbound_sockets            | 127.9.176.195                                                                                                                                                   | experimental          | 127.9.176.195:41211
 local_ip_for_outbound_sockets            | 127.9.176.196                                                                                                                                                   | experimental          | 127.9.176.196:41329
 local_ip_for_outbound_sockets            | 127.9.176.197                                                                                                                                                   | experimental          | 127.9.176.197:35757
 local_ip_for_outbound_sockets            | 127.9.176.198                                                                                                                                                   | experimental          | 127.9.176.198:38527
 local_ip_for_outbound_sockets            | 127.9.176.199                                                                                                                                                   | experimental          | 127.9.176.199:38645
 local_ip_for_outbound_sockets            | 127.9.176.200                                                                                                                                                   | experimental          | 127.9.176.200:42345
 never_fsync                              | true                                                                                                                                                            | unsafe,advanced       | all 8 server(s) checked
 openssl_security_level_override          | 0                                                                                                                                                               | unsafe,hidden         | all 8 server(s) checked
 raft_prepare_replacement_before_eviction | true                                                                                                                                                            | experimental,advanced | all 8 server(s) checked
 rpc_server_allow_ephemeral_ports         | true                                                                                                                                                            | unsafe                | all 8 server(s) checked
 server_dump_info_format                  | pb                                                                                                                                                              | hidden                | all 8 server(s) checked
 server_dump_info_path                    | /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden                | 127.9.176.193:33849
 server_dump_info_path                    | /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden                | 127.9.176.194:34851
 server_dump_info_path                    | /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden                | 127.9.176.195:41211
 server_dump_info_path                    | /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden                | 127.9.176.196:41329
 server_dump_info_path                    | /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden                | 127.9.176.197:35757
 server_dump_info_path                    | /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden                | 127.9.176.198:38527
 server_dump_info_path                    | /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden                | 127.9.176.199:38645
 server_dump_info_path                    | /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-7/data/info.pb | hidden                | 127.9.176.200:42345

Flags of checked categories for Tablet Server:
        Flag         |        Value        |      Tablet Server
---------------------+---------------------+-------------------------
 builtin_ntp_servers | 127.9.176.212:33337 | all 8 server(s) checked
 time_source         | builtin             | all 8 server(s) checked

Version Summary
     Version     |         Servers
-----------------+-------------------------
 1.18.0-SNAPSHOT | all 9 server(s) checked

Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
          Name          | RF | Status  | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
 TestTable              | 2  | HEALTHY | 1             | 1       | 0          | 0                | 0
 rebalance_test_table_0 | 2  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_1 | 2  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_2 | 2  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_3 | 2  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_4 | 2  | HEALTHY | 3             | 3       | 0          | 0                | 0

Tablet Replica Count Summary
   Statistic    | Replica Count
----------------+---------------
 Minimum        | 4
 First Quartile | 4
 Median         | 4
 Third Quartile | 4
 Maximum        | 4

Total Count Summary
                | Total Count
----------------+-------------
 Masters        | 1
 Tablet Servers | 8
 Tables         | 6
 Tablets        | 16
 Replicas       | 32

==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set

OK
I20240502 01:53:55.553344  9923 log_verifier.cc:126] Checking tablet 01fb93c5dab3401fa5a103ead4dd9708
I20240502 01:53:55.680379  9923 log_verifier.cc:177] Verified matching terms for 349 ops in tablet 01fb93c5dab3401fa5a103ead4dd9708
I20240502 01:53:55.680878  9923 log_verifier.cc:126] Checking tablet 4c0ab7c2995d4ccfa170f365b4e09b1d
I20240502 01:53:55.816689  9923 log_verifier.cc:177] Verified matching terms for 332 ops in tablet 4c0ab7c2995d4ccfa170f365b4e09b1d
I20240502 01:53:55.817193  9923 log_verifier.cc:126] Checking tablet 57765e8e3ff7438ca9e588db8e685a03
I20240502 01:53:55.963976  9923 log_verifier.cc:177] Verified matching terms for 339 ops in tablet 57765e8e3ff7438ca9e588db8e685a03
I20240502 01:53:55.964516  9923 log_verifier.cc:126] Checking tablet 6682ed5cba8448c7b3cf74ab726a18e7
I20240502 01:53:56.093636  9923 log_verifier.cc:177] Verified matching terms for 306 ops in tablet 6682ed5cba8448c7b3cf74ab726a18e7
I20240502 01:53:56.094074  9923 log_verifier.cc:126] Checking tablet 71b0f9fbee82464994252126b2d631bc
I20240502 01:53:56.213832  9923 log_verifier.cc:177] Verified matching terms for 297 ops in tablet 71b0f9fbee82464994252126b2d631bc
I20240502 01:53:56.214241  9923 log_verifier.cc:126] Checking tablet 8477215e8d4043c581c4ce7e5aa0fbd6
I20240502 01:53:56.322167  9923 log_verifier.cc:177] Verified matching terms for 293 ops in tablet 8477215e8d4043c581c4ce7e5aa0fbd6
I20240502 01:53:56.322575  9923 log_verifier.cc:126] Checking tablet 866d771c178a450e84189c64ddfca9d7
I20240502 01:53:56.434432  9923 log_verifier.cc:177] Verified matching terms for 298 ops in tablet 866d771c178a450e84189c64ddfca9d7
I20240502 01:53:56.434850  9923 log_verifier.cc:126] Checking tablet 930645182ab748d7b864fece1e858b3f
I20240502 01:53:56.546859  9923 log_verifier.cc:177] Verified matching terms for 294 ops in tablet 930645182ab748d7b864fece1e858b3f
I20240502 01:53:56.547271  9923 log_verifier.cc:126] Checking tablet b38a1fb4d1694c609a932e8816fc3514
I20240502 01:53:56.693660  9923 log_verifier.cc:177] Verified matching terms for 318 ops in tablet b38a1fb4d1694c609a932e8816fc3514
I20240502 01:53:56.694223  9923 log_verifier.cc:126] Checking tablet bdf6095f84aa497ba5db95d55c010085
I20240502 01:53:56.817531  9923 log_verifier.cc:177] Verified matching terms for 315 ops in tablet bdf6095f84aa497ba5db95d55c010085
I20240502 01:53:56.817938  9923 log_verifier.cc:126] Checking tablet d38a05f7ffa94cc5abc50a76a83692b5
I20240502 01:53:56.829913  9923 log_verifier.cc:177] Verified matching terms for 5 ops in tablet d38a05f7ffa94cc5abc50a76a83692b5
I20240502 01:53:56.830005  9923 log_verifier.cc:126] Checking tablet d4511dd1019c4bdcbf9fc7e4bfe2afb7
I20240502 01:53:56.953702  9923 log_verifier.cc:177] Verified matching terms for 308 ops in tablet d4511dd1019c4bdcbf9fc7e4bfe2afb7
I20240502 01:53:56.954109  9923 log_verifier.cc:126] Checking tablet daed3123a96a496da83a6110e6943264
I20240502 01:53:57.075196  9923 log_verifier.cc:177] Verified matching terms for 329 ops in tablet daed3123a96a496da83a6110e6943264
I20240502 01:53:57.075693  9923 log_verifier.cc:126] Checking tablet eb1cf83294e044b49e115323de67ed42
I20240502 01:53:57.214991  9923 log_verifier.cc:177] Verified matching terms for 324 ops in tablet eb1cf83294e044b49e115323de67ed42
I20240502 01:53:57.215394  9923 log_verifier.cc:126] Checking tablet edf0aa94bfb94d499015897713ed94fa
I20240502 01:53:57.327137  9923 log_verifier.cc:177] Verified matching terms for 306 ops in tablet edf0aa94bfb94d499015897713ed94fa
I20240502 01:53:57.327729  9923 log_verifier.cc:126] Checking tablet f2af8f5d5e784e1988a957666844db6b
I20240502 01:53:57.439971  9923 log_verifier.cc:177] Verified matching terms for 298 ops in tablet f2af8f5d5e784e1988a957666844db6b
I20240502 01:53:57.449745  9923 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu pid 13387
I20240502 01:53:57.450893 13511 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:53:57.690615  9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu with pid 13387
W20240502 01:53:57.735173 13538 connection.cc:535] server connection from 127.9.176.193:59749 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 01:53:57.735615  9923 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu pid 13521
W20240502 01:53:57.735999 13311 connection.cc:535] server connection from 127.9.176.193:51527 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 01:53:57.736693 13645 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:53:57.963019  9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu with pid 13521
W20240502 01:53:58.005839 14751 connection.cc:535] server connection from 127.9.176.194:33813 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:53:58.005860 15019 connection.cc:535] server connection from 127.9.176.194:34033 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 01:53:58.006284  9923 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu pid 13653
I20240502 01:53:58.007313 13775 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:53:58.231119  9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu with pid 13653
W20240502 01:53:58.275698 15019 connection.cc:535] server connection from 127.9.176.195:35639 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 01:53:58.275995  9923 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu pid 14736
I20240502 01:53:58.283329 14862 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:53:58.438778  9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu with pid 14736
I20240502 01:53:58.470080  9923 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu pid 14871
I20240502 01:53:58.476500 14996 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:53:58.641044  9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu with pid 14871
I20240502 01:53:58.674049  9923 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu pid 15004
I20240502 01:53:58.681094 15127 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:53:58.861430  9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu with pid 15004
W20240502 01:53:58.872481 16569 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.176.196:41329: connect: Connection refused (error 111)
W20240502 01:53:58.874269 16569 consensus_peers.cc:485] T bdf6095f84aa497ba5db95d55c010085 P a82548707a5844ce9650f2e9dbe4f581 -> Peer ce90a4bd06ba41dd8e5aa227de9780a5 (127.9.176.196:41329): Couldn't send request to peer ce90a4bd06ba41dd8e5aa227de9780a5. Status: Network error: Client connection negotiation failed: client connection to 127.9.176.196:41329: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:58.894572  9923 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu pid 15136
I20240502 01:53:58.900303 15267 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:53:59.057245  9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu with pid 15136
I20240502 01:53:59.088375  9923 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu pid 16552
I20240502 01:53:59.094260 16677 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:53:59.221825 17291 raft_consensus.cc:482] T 6682ed5cba8448c7b3cf74ab726a18e7 P a82548707a5844ce9650f2e9dbe4f581 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5359629f948641f4b56e35ad6b6a15c4)
I20240502 01:53:59.222075 17291 raft_consensus.cc:504] T 6682ed5cba8448c7b3cf74ab726a18e7 P a82548707a5844ce9650f2e9dbe4f581 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 306 OBSOLETE_local: false peers { permanent_uuid: "5359629f948641f4b56e35ad6b6a15c4" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33849 } } peers { permanent_uuid: "a82548707a5844ce9650f2e9dbe4f581" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 42345 } attrs { promote: false } }
I20240502 01:53:59.222774 17291 leader_election.cc:288] T 6682ed5cba8448c7b3cf74ab726a18e7 P a82548707a5844ce9650f2e9dbe4f581 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5359629f948641f4b56e35ad6b6a15c4 (127.9.176.193:33849)
I20240502 01:53:59.223027  9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu with pid 16552
I20240502 01:53:59.245965  9923 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu pid 13295
I20240502 01:53:59.247087 13355 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:53:59.421772  9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu with pid 13295
2024-05-02T01:53:59Z chronyd exiting
[       OK ] RebalanceParamTest.Rebalance/3 (50904 ms)
[----------] 1 test from RebalanceParamTest (50904 ms total)

[----------] 1 test from ConcurrentRebalancersTest
[ RUN      ] ConcurrentRebalancersTest.TwoConcurrentRebalancers/1
I20240502 01:53:59.530323  9923 test_util.cc:268] Using random seed: -529542333
I20240502 01:53:59.532222  9923 ts_itest-base.cc:112] Starting cluster with:
I20240502 01:53:59.532310  9923 ts_itest-base.cc:113] --------------
I20240502 01:53:59.532338  9923 ts_itest-base.cc:114] 8 tablet servers
I20240502 01:53:59.532364  9923 ts_itest-base.cc:115] 3 replicas per TS
I20240502 01:53:59.532397  9923 ts_itest-base.cc:116] --------------
2024-05-02T01:53:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T01:53:59Z Disabled control of system clock
I20240502 01:53:59.590668  9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
/tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:46743
--webserver_interface=127.9.176.254
--webserver_port=0
--builtin_ntp_servers=127.9.176.212:41499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.176.254:46743
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240502 01:53:59.972796 17325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:59.973157 17325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:53:59.973302 17325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:59.981469 17325 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:53:59.981623 17325 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 01:53:59.981711 17325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:53:59.981804 17325 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 01:53:59.981876 17325 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 01:53:59.992483 17325 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:41499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.9.176.254:46743
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:46743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.9.176.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true

Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:53:59.994436 17325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:53:59.997148 17325 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:54:00.009873 17331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:00.010468 17333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:00.010991 17330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:54:00.011708 17325 server_base.cc:1034] running on GCE node
I20240502 01:54:00.012639 17325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:54:00.014266 17325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:54:00.015506 17325 hybrid_clock.cc:648] HybridClock initialized: now 1714614840015446 us; error 66 us; skew 500 ppm
I20240502 01:54:00.015897 17325 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:54:00.023794 17325 webserver.cc:460] Webserver started at http://127.9.176.254:45083/ using document root <none> and password file <none>
I20240502 01:54:00.024698 17325 fs_manager.cc:359] Metadata directory not provided
I20240502 01:54:00.024848 17325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:54:00.025238 17325 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:54:00.028313 17325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "3f38232389884c2aa48edf3d5f0f1fd2"
format_stamp: "Formatted at 2024-05-02 01:54:00 on dist-test-slave-j3hh"
I20240502 01:54:00.029208 17325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "3f38232389884c2aa48edf3d5f0f1fd2"
format_stamp: "Formatted at 2024-05-02 01:54:00 on dist-test-slave-j3hh"
I20240502 01:54:00.035389 17325 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.006s	sys 0.001s
I20240502 01:54:00.039309 17339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:54:00.040899 17325 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240502 01:54:00.041097 17325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/wal
uuid: "3f38232389884c2aa48edf3d5f0f1fd2"
format_stamp: "Formatted at 2024-05-02 01:54:00 on dist-test-slave-j3hh"
I20240502 01:54:00.041342 17325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:54:00.065951 17325 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:54:00.066954 17325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:54:00.067291 17325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:54:00.088618 17325 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.254:46743
I20240502 01:54:00.088635 17390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.254:46743 every 8 connection(s)
I20240502 01:54:00.090273 17325 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 01:54:00.092120  9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu as pid 17325
I20240502 01:54:00.092347  9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 01:54:00.095113 17391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:00.107782 17391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f38232389884c2aa48edf3d5f0f1fd2: Bootstrap starting.
I20240502 01:54:00.111364 17391 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f38232389884c2aa48edf3d5f0f1fd2: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:00.112850 17391 log.cc:824] T 00000000000000000000000000000000 P 3f38232389884c2aa48edf3d5f0f1fd2: Log is configured to *not* fsync() on all Append() calls
I20240502 01:54:00.116048 17391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f38232389884c2aa48edf3d5f0f1fd2: No bootstrap required, opened a new log
I20240502 01:54:00.124066 17391 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f38232389884c2aa48edf3d5f0f1fd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f38232389884c2aa48edf3d5f0f1fd2" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 46743 } }
I20240502 01:54:00.124491 17391 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3f38232389884c2aa48edf3d5f0f1fd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:00.124608 17391 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f38232389884c2aa48edf3d5f0f1fd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f38232389884c2aa48edf3d5f0f1fd2, State: Initialized, Role: FOLLOWER
I20240502 01:54:00.125313 17391 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f38232389884c2aa48edf3d5f0f1fd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f38232389884c2aa48edf3d5f0f1fd2" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 46743 } }
I20240502 01:54:00.125576 17391 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f38232389884c2aa48edf3d5f0f1fd2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:54:00.125712 17391 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f38232389884c2aa48edf3d5f0f1fd2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:54:00.125874 17391 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f38232389884c2aa48edf3d5f0f1fd2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:00.128219 17391 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f38232389884c2aa48edf3d5f0f1fd2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f38232389884c2aa48edf3d5f0f1fd2" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 46743 } }
I20240502 01:54:00.128804 17391 leader_election.cc:302] T 00000000000000000000000000000000 P 3f38232389884c2aa48edf3d5f0f1fd2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f38232389884c2aa48edf3d5f0f1fd2; no voters: 
I20240502 01:54:00.129438 17391 leader_election.cc:288] T 00000000000000000000000000000000 P 3f38232389884c2aa48edf3d5f0f1fd2 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240502 01:54:00.129598 17396 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f38232389884c2aa48edf3d5f0f1fd2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:00.131275 17396 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f38232389884c2aa48edf3d5f0f1fd2 [term 1 LEADER]: Becoming Leader. State: Replica: 3f38232389884c2aa48edf3d5f0f1fd2, State: Running, Role: LEADER
I20240502 01:54:00.132019 17391 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f38232389884c2aa48edf3d5f0f1fd2 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 01:54:00.131981 17396 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f38232389884c2aa48edf3d5f0f1fd2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f38232389884c2aa48edf3d5f0f1fd2" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 46743 } }
I20240502 01:54:00.134948 17398 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f38232389884c2aa48edf3d5f0f1fd2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f38232389884c2aa48edf3d5f0f1fd2. Latest consensus state: current_term: 1 leader_uuid: "3f38232389884c2aa48edf3d5f0f1fd2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f38232389884c2aa48edf3d5f0f1fd2" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 46743 } } }
I20240502 01:54:00.135269 17398 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f38232389884c2aa48edf3d5f0f1fd2 [sys.catalog]: This master's current role is: LEADER
I20240502 01:54:00.135069 17397 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f38232389884c2aa48edf3d5f0f1fd2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3f38232389884c2aa48edf3d5f0f1fd2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f38232389884c2aa48edf3d5f0f1fd2" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 46743 } } }
I20240502 01:54:00.135506 17397 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f38232389884c2aa48edf3d5f0f1fd2 [sys.catalog]: This master's current role is: LEADER
W20240502 01:54:00.149832 17413 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3f38232389884c2aa48edf3d5f0f1fd2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240502 01:54:00.149987 17413 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240502 01:54:00.150208 17404 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 01:54:00.152738 17404 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 01:54:00.161880 17404 catalog_manager.cc:1347] Generated new cluster ID: 912040282974486d98069eefef0f2732
I20240502 01:54:00.162050 17404 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 01:54:00.182173 17404 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 01:54:00.183606 17404 catalog_manager.cc:1504] Loading token signing keys...
I20240502 01:54:00.208226 17404 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 3f38232389884c2aa48edf3d5f0f1fd2: Generated new TSK 0
I20240502 01:54:00.209167 17404 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 01:54:00.214126  9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
/tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.193:0
--local_ip_for_outbound_sockets=127.9.176.193
--webserver_interface=127.9.176.193
--webserver_port=0
--tserver_master_addrs=127.9.176.254:46743
--builtin_ntp_servers=127.9.176.212:41499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 01:54:00.588223 17416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:54:00.588557 17416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:54:00.588651 17416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:54:00.596669 17416 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:54:00.596823 17416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:54:00.596953 17416 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.193
I20240502 01:54:00.607724 17416 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:41499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.9.176.193
--webserver_port=0
--tserver_master_addrs=127.9.176.254:46743
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.193
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:54:00.609709 17416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:54:00.612283 17416 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:54:00.625244 17422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:00.625806 17425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:00.626345 17421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:54:00.626924 17416 server_base.cc:1034] running on GCE node
I20240502 01:54:00.627557 17416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:54:00.628638 17416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:54:00.629855 17416 hybrid_clock.cc:648] HybridClock initialized: now 1714614840629812 us; error 43 us; skew 500 ppm
I20240502 01:54:00.630231 17416 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:54:00.632858 17416 webserver.cc:460] Webserver started at http://127.9.176.193:40777/ using document root <none> and password file <none>
I20240502 01:54:00.633731 17416 fs_manager.cc:359] Metadata directory not provided
I20240502 01:54:00.633872 17416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:54:00.634218 17416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:54:00.637157 17416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "311f765f59754bdcb4eadc04e14bd4ea"
format_stamp: "Formatted at 2024-05-02 01:54:00 on dist-test-slave-j3hh"
I20240502 01:54:00.638021 17416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "311f765f59754bdcb4eadc04e14bd4ea"
format_stamp: "Formatted at 2024-05-02 01:54:00 on dist-test-slave-j3hh"
I20240502 01:54:00.644433 17416 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.003s	sys 0.004s
I20240502 01:54:00.648597 17431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:54:00.650142 17416 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.004s
I20240502 01:54:00.650338 17416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "311f765f59754bdcb4eadc04e14bd4ea"
format_stamp: "Formatted at 2024-05-02 01:54:00 on dist-test-slave-j3hh"
I20240502 01:54:00.650588 17416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:54:00.674362 17416 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:54:00.675335 17416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:54:00.675709 17416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:54:00.676811 17416 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:54:00.678565 17416 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:54:00.678678 17416 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:54:00.678819 17416 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:54:00.678897 17416 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:54:00.713784 17416 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.193:35397
I20240502 01:54:00.713835 17544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.193:35397 every 8 connection(s)
I20240502 01:54:00.715214 17416 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 01:54:00.716428  9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu as pid 17416
I20240502 01:54:00.716646  9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 01:54:00.720291  9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
/tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.194:0
--local_ip_for_outbound_sockets=127.9.176.194
--webserver_interface=127.9.176.194
--webserver_port=0
--tserver_master_addrs=127.9.176.254:46743
--builtin_ntp_servers=127.9.176.212:41499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:54:00.730151 17545 heartbeater.cc:347] Connected to a master server at 127.9.176.254:46743
I20240502 01:54:00.730703 17545 heartbeater.cc:464] Registering TS with master...
I20240502 01:54:00.731752 17545 heartbeater.cc:511] Master 127.9.176.254:46743 requested a full tablet report, sending...
I20240502 01:54:00.734697 17356 ts_manager.cc:194] Registered new tserver with Master: 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193:35397)
I20240502 01:54:00.736241 17356 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.176.193:40371
W20240502 01:54:01.083478 17549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:54:01.083920 17549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:54:01.084028 17549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:54:01.092181 17549 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:54:01.092363 17549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:54:01.092556 17549 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.194
I20240502 01:54:01.102427 17549 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:41499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.9.176.194
--webserver_port=0
--tserver_master_addrs=127.9.176.254:46743
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.194
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:54:01.103902 17549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:54:01.105947 17549 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:54:01.118885 17554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:01.119484 17558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:01.120059 17555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:54:01.120761 17549 server_base.cc:1034] running on GCE node
I20240502 01:54:01.121377 17549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:54:01.122370 17549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:54:01.123603 17549 hybrid_clock.cc:648] HybridClock initialized: now 1714614841123514 us; error 48 us; skew 500 ppm
I20240502 01:54:01.124020 17549 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:54:01.127192 17549 webserver.cc:460] Webserver started at http://127.9.176.194:38703/ using document root <none> and password file <none>
I20240502 01:54:01.128016 17549 fs_manager.cc:359] Metadata directory not provided
I20240502 01:54:01.128121 17549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:54:01.128415 17549 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:54:01.131132 17549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "1b50a031f2784ba498b3e52e16916205"
format_stamp: "Formatted at 2024-05-02 01:54:01 on dist-test-slave-j3hh"
I20240502 01:54:01.131939 17549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "1b50a031f2784ba498b3e52e16916205"
format_stamp: "Formatted at 2024-05-02 01:54:01 on dist-test-slave-j3hh"
I20240502 01:54:01.137554 17549 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.004s	sys 0.000s
I20240502 01:54:01.141300 17564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:54:01.142761 17549 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.000s
I20240502 01:54:01.142979 17549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "1b50a031f2784ba498b3e52e16916205"
format_stamp: "Formatted at 2024-05-02 01:54:01 on dist-test-slave-j3hh"
I20240502 01:54:01.143204 17549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:54:01.163154 17549 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:54:01.164186 17549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:54:01.164494 17549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:54:01.165609 17549 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:54:01.167219 17549 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:54:01.167327 17549 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:54:01.167472 17549 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:54:01.167548 17549 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:54:01.202420 17549 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.194:45617
I20240502 01:54:01.202476 17680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.194:45617 every 8 connection(s)
I20240502 01:54:01.203869 17549 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 01:54:01.208227  9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu as pid 17549
I20240502 01:54:01.208434  9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 01:54:01.211764  9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
/tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.195:0
--local_ip_for_outbound_sockets=127.9.176.195
--webserver_interface=127.9.176.195
--webserver_port=0
--tserver_master_addrs=127.9.176.254:46743
--builtin_ntp_servers=127.9.176.212:41499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:54:01.218050 17681 heartbeater.cc:347] Connected to a master server at 127.9.176.254:46743
I20240502 01:54:01.218434 17681 heartbeater.cc:464] Registering TS with master...
I20240502 01:54:01.219321 17681 heartbeater.cc:511] Master 127.9.176.254:46743 requested a full tablet report, sending...
I20240502 01:54:01.221083 17356 ts_manager.cc:194] Registered new tserver with Master: 1b50a031f2784ba498b3e52e16916205 (127.9.176.194:45617)
I20240502 01:54:01.222278 17356 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.176.194:59767
W20240502 01:54:01.576516 17684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:54:01.576846 17684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:54:01.576961 17684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:54:01.584733 17684 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:54:01.584889 17684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:54:01.585032 17684 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.195
I20240502 01:54:01.594530 17684 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:41499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.9.176.195
--webserver_port=0
--tserver_master_addrs=127.9.176.254:46743
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.195
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:54:01.595991 17684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:54:01.597852 17684 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:54:01.610908 17689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:01.611532 17690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:01.612186 17692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:54:01.612886 17684 server_base.cc:1034] running on GCE node
I20240502 01:54:01.613575 17684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:54:01.614571 17684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:54:01.615788 17684 hybrid_clock.cc:648] HybridClock initialized: now 1714614841615741 us; error 38 us; skew 500 ppm
I20240502 01:54:01.616150 17684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:54:01.618652 17684 webserver.cc:460] Webserver started at http://127.9.176.195:40149/ using document root <none> and password file <none>
I20240502 01:54:01.619470 17684 fs_manager.cc:359] Metadata directory not provided
I20240502 01:54:01.619632 17684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:54:01.619988 17684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:54:01.622812 17684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "84d3a4b174c34e02aedab1e620731359"
format_stamp: "Formatted at 2024-05-02 01:54:01 on dist-test-slave-j3hh"
I20240502 01:54:01.623765 17684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "84d3a4b174c34e02aedab1e620731359"
format_stamp: "Formatted at 2024-05-02 01:54:01 on dist-test-slave-j3hh"
I20240502 01:54:01.629582 17684 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.004s	sys 0.000s
I20240502 01:54:01.633647 17698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:54:01.635254 17684 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20240502 01:54:01.635463 17684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "84d3a4b174c34e02aedab1e620731359"
format_stamp: "Formatted at 2024-05-02 01:54:01 on dist-test-slave-j3hh"
I20240502 01:54:01.635735 17684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:54:01.655251 17684 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:54:01.656332 17684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:54:01.656644 17684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:54:01.657748 17684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:54:01.659466 17684 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:54:01.659615 17684 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:54:01.659754 17684 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:54:01.659838 17684 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:54:01.697613 17684 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.195:37357
I20240502 01:54:01.697669 17813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.195:37357 every 8 connection(s)
I20240502 01:54:01.699079 17684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 01:54:01.708793  9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu as pid 17684
I20240502 01:54:01.709022  9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 01:54:01.713019  9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
/tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.196:0
--local_ip_for_outbound_sockets=127.9.176.196
--webserver_interface=127.9.176.196
--webserver_port=0
--tserver_master_addrs=127.9.176.254:46743
--builtin_ntp_servers=127.9.176.212:41499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:54:01.714497 17814 heartbeater.cc:347] Connected to a master server at 127.9.176.254:46743
I20240502 01:54:01.714856 17814 heartbeater.cc:464] Registering TS with master...
I20240502 01:54:01.716043 17814 heartbeater.cc:511] Master 127.9.176.254:46743 requested a full tablet report, sending...
I20240502 01:54:01.717994 17356 ts_manager.cc:194] Registered new tserver with Master: 84d3a4b174c34e02aedab1e620731359 (127.9.176.195:37357)
I20240502 01:54:01.718883 17356 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.176.195:60083
I20240502 01:54:01.740093 17545 heartbeater.cc:503] Master 127.9.176.254:46743 was elected leader, sending a full tablet report...
W20240502 01:54:02.097180 17817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:54:02.097661 17817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:54:02.097827 17817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:54:02.106670 17817 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:54:02.106819 17817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:54:02.106951 17817 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.196
I20240502 01:54:02.116526 17817 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:41499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.9.176.196
--webserver_port=0
--tserver_master_addrs=127.9.176.254:46743
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.196
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:54:02.117851 17817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:54:02.119673 17817 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:54:02.133581 17823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:02.134011 17825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:02.134652 17822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:54:02.135298 17817 server_base.cc:1034] running on GCE node
I20240502 01:54:02.135967 17817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:54:02.136988 17817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:54:02.138260 17817 hybrid_clock.cc:648] HybridClock initialized: now 1714614842138212 us; error 44 us; skew 500 ppm
I20240502 01:54:02.138597 17817 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:54:02.141311 17817 webserver.cc:460] Webserver started at http://127.9.176.196:43089/ using document root <none> and password file <none>
I20240502 01:54:02.142138 17817 fs_manager.cc:359] Metadata directory not provided
I20240502 01:54:02.142253 17817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:54:02.142567 17817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:54:02.145588 17817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "da3584f429104d23a1a8899b35cc2138"
format_stamp: "Formatted at 2024-05-02 01:54:02 on dist-test-slave-j3hh"
I20240502 01:54:02.146554 17817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "da3584f429104d23a1a8899b35cc2138"
format_stamp: "Formatted at 2024-05-02 01:54:02 on dist-test-slave-j3hh"
I20240502 01:54:02.152807 17817 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.006s	sys 0.001s
I20240502 01:54:02.156733 17831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:54:02.158499 17817 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.001s
I20240502 01:54:02.158710 17817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "da3584f429104d23a1a8899b35cc2138"
format_stamp: "Formatted at 2024-05-02 01:54:02 on dist-test-slave-j3hh"
I20240502 01:54:02.158938 17817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:54:02.177201 17817 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:54:02.178169 17817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:54:02.178478 17817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:54:02.179543 17817 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:54:02.181336 17817 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:54:02.181437 17817 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:54:02.181528 17817 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:54:02.181584 17817 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:54:02.216497 17817 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.196:46381
I20240502 01:54:02.216555 17946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.196:46381 every 8 connection(s)
I20240502 01:54:02.217862 17817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 01:54:02.222554  9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu as pid 17817
I20240502 01:54:02.222745  9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 01:54:02.225239 17681 heartbeater.cc:503] Master 127.9.176.254:46743 was elected leader, sending a full tablet report...
I20240502 01:54:02.226753  9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
/tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.197:0
--local_ip_for_outbound_sockets=127.9.176.197
--webserver_interface=127.9.176.197
--webserver_port=0
--tserver_master_addrs=127.9.176.254:46743
--builtin_ntp_servers=127.9.176.212:41499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:54:02.232846 17947 heartbeater.cc:347] Connected to a master server at 127.9.176.254:46743
I20240502 01:54:02.233199 17947 heartbeater.cc:464] Registering TS with master...
I20240502 01:54:02.234151 17947 heartbeater.cc:511] Master 127.9.176.254:46743 requested a full tablet report, sending...
I20240502 01:54:02.235738 17356 ts_manager.cc:194] Registered new tserver with Master: da3584f429104d23a1a8899b35cc2138 (127.9.176.196:46381)
I20240502 01:54:02.236529 17356 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.176.196:60471
W20240502 01:54:02.592964 17951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:54:02.593298 17951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:54:02.593396 17951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:54:02.601122 17951 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:54:02.601266 17951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:54:02.601384 17951 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.197
I20240502 01:54:02.611145 17951 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:41499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.197:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.9.176.197
--webserver_port=0
--tserver_master_addrs=127.9.176.254:46743
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.197
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:54:02.612514 17951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:54:02.614200 17951 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:54:02.626788 17956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:02.627390 17957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:02.627918 17959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:54:02.628634 17951 server_base.cc:1034] running on GCE node
I20240502 01:54:02.629271 17951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:54:02.630371 17951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:54:02.631634 17951 hybrid_clock.cc:648] HybridClock initialized: now 1714614842631546 us; error 46 us; skew 500 ppm
I20240502 01:54:02.632058 17951 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:54:02.634495 17951 webserver.cc:460] Webserver started at http://127.9.176.197:39411/ using document root <none> and password file <none>
I20240502 01:54:02.635289 17951 fs_manager.cc:359] Metadata directory not provided
I20240502 01:54:02.635411 17951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:54:02.635776 17951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:54:02.638538 17951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "29806d3d37354c169484f6db368f8798"
format_stamp: "Formatted at 2024-05-02 01:54:02 on dist-test-slave-j3hh"
I20240502 01:54:02.639461 17951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "29806d3d37354c169484f6db368f8798"
format_stamp: "Formatted at 2024-05-02 01:54:02 on dist-test-slave-j3hh"
I20240502 01:54:02.645666 17951 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.004s	sys 0.000s
I20240502 01:54:02.649852 17965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:54:02.651412 17951 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20240502 01:54:02.651634 17951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "29806d3d37354c169484f6db368f8798"
format_stamp: "Formatted at 2024-05-02 01:54:02 on dist-test-slave-j3hh"
I20240502 01:54:02.651854 17951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:54:02.675881 17951 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:54:02.676800 17951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:54:02.677074 17951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:54:02.678110 17951 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:54:02.679905 17951 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:54:02.680013 17951 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:54:02.680110 17951 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:54:02.680176 17951 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:54:02.715647 17951 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.197:35011
I20240502 01:54:02.715703 18080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.197:35011 every 8 connection(s)
I20240502 01:54:02.717459 17951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 01:54:02.721810 17814 heartbeater.cc:503] Master 127.9.176.254:46743 was elected leader, sending a full tablet report...
I20240502 01:54:02.724718  9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu as pid 17951
I20240502 01:54:02.724906  9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 01:54:02.728193  9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
/tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.198:0
--local_ip_for_outbound_sockets=127.9.176.198
--webserver_interface=127.9.176.198
--webserver_port=0
--tserver_master_addrs=127.9.176.254:46743
--builtin_ntp_servers=127.9.176.212:41499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:54:02.741940 18081 heartbeater.cc:347] Connected to a master server at 127.9.176.254:46743
I20240502 01:54:02.742524 18081 heartbeater.cc:464] Registering TS with master...
I20240502 01:54:02.743947 18081 heartbeater.cc:511] Master 127.9.176.254:46743 requested a full tablet report, sending...
I20240502 01:54:02.745752 17356 ts_manager.cc:194] Registered new tserver with Master: 29806d3d37354c169484f6db368f8798 (127.9.176.197:35011)
I20240502 01:54:02.746675 17356 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.176.197:37437
W20240502 01:54:03.103165 18084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:54:03.103503 18084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:54:03.103652 18084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:54:03.111753 18084 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:54:03.111904 18084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:54:03.112085 18084 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.198
I20240502 01:54:03.121840 18084 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:41499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.198:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.9.176.198
--webserver_port=0
--tserver_master_addrs=127.9.176.254:46743
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.198
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:54:03.123227 18084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:54:03.125038 18084 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:54:03.138679 18089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:03.139308 18090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:03.139880 18092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:54:03.140590 18084 server_base.cc:1034] running on GCE node
I20240502 01:54:03.141317 18084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:54:03.142413 18084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:54:03.143662 18084 hybrid_clock.cc:648] HybridClock initialized: now 1714614843143603 us; error 54 us; skew 500 ppm
I20240502 01:54:03.144011 18084 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:54:03.146682 18084 webserver.cc:460] Webserver started at http://127.9.176.198:40127/ using document root <none> and password file <none>
I20240502 01:54:03.147624 18084 fs_manager.cc:359] Metadata directory not provided
I20240502 01:54:03.147760 18084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:54:03.148133 18084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:54:03.151095 18084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "4b7c382c9edd41babdd1a1afc904a813"
format_stamp: "Formatted at 2024-05-02 01:54:03 on dist-test-slave-j3hh"
I20240502 01:54:03.152031 18084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "4b7c382c9edd41babdd1a1afc904a813"
format_stamp: "Formatted at 2024-05-02 01:54:03 on dist-test-slave-j3hh"
I20240502 01:54:03.158406 18084 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.002s	sys 0.003s
I20240502 01:54:03.162626 18098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:54:03.164393 18084 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.000s
I20240502 01:54:03.164597 18084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "4b7c382c9edd41babdd1a1afc904a813"
format_stamp: "Formatted at 2024-05-02 01:54:03 on dist-test-slave-j3hh"
I20240502 01:54:03.164830 18084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:54:03.184711 18084 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:54:03.185637 18084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:54:03.185935 18084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:54:03.186962 18084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:54:03.188709 18084 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:54:03.188812 18084 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:54:03.188905 18084 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:54:03.188962 18084 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:54:03.224690 18084 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.198:36865
I20240502 01:54:03.224743 18212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.198:36865 every 8 connection(s)
I20240502 01:54:03.226006 18084 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 01:54:03.229642  9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu as pid 18084
I20240502 01:54:03.229820  9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240502 01:54:03.233067  9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
/tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.199:0
--local_ip_for_outbound_sockets=127.9.176.199
--webserver_interface=127.9.176.199
--webserver_port=0
--tserver_master_addrs=127.9.176.254:46743
--builtin_ntp_servers=127.9.176.212:41499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:54:03.239133 17947 heartbeater.cc:503] Master 127.9.176.254:46743 was elected leader, sending a full tablet report...
I20240502 01:54:03.239976 18213 heartbeater.cc:347] Connected to a master server at 127.9.176.254:46743
I20240502 01:54:03.240404 18213 heartbeater.cc:464] Registering TS with master...
I20240502 01:54:03.241714 18213 heartbeater.cc:511] Master 127.9.176.254:46743 requested a full tablet report, sending...
I20240502 01:54:03.243294 17356 ts_manager.cc:194] Registered new tserver with Master: 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198:36865)
I20240502 01:54:03.244207 17356 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.176.198:42633
W20240502 01:54:03.601361 18216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:54:03.601691 18216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:54:03.601821 18216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:54:03.609676 18216 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:54:03.609834 18216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:54:03.610060 18216 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.199
I20240502 01:54:03.619922 18216 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:41499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.199:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.9.176.199
--webserver_port=0
--tserver_master_addrs=127.9.176.254:46743
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.199
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:54:03.621273 18216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:54:03.623099 18216 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:54:03.636751 18223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:03.637313 18225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:03.637934 18222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:54:03.638592 18216 server_base.cc:1034] running on GCE node
I20240502 01:54:03.639235 18216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:54:03.640350 18216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:54:03.641567 18216 hybrid_clock.cc:648] HybridClock initialized: now 1714614843641498 us; error 71 us; skew 500 ppm
I20240502 01:54:03.641934 18216 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:54:03.644623 18216 webserver.cc:460] Webserver started at http://127.9.176.199:45347/ using document root <none> and password file <none>
I20240502 01:54:03.645474 18216 fs_manager.cc:359] Metadata directory not provided
I20240502 01:54:03.645619 18216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:54:03.646003 18216 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:54:03.649176 18216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "272b2179c1b4476090a89880df2a8baa"
format_stamp: "Formatted at 2024-05-02 01:54:03 on dist-test-slave-j3hh"
I20240502 01:54:03.650048 18216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "272b2179c1b4476090a89880df2a8baa"
format_stamp: "Formatted at 2024-05-02 01:54:03 on dist-test-slave-j3hh"
I20240502 01:54:03.656507 18216 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.007s	sys 0.000s
I20240502 01:54:03.660926 18231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:54:03.662675 18216 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.001s	sys 0.004s
I20240502 01:54:03.662923 18216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "272b2179c1b4476090a89880df2a8baa"
format_stamp: "Formatted at 2024-05-02 01:54:03 on dist-test-slave-j3hh"
I20240502 01:54:03.663179 18216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:54:03.687372 18216 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:54:03.688372 18216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:54:03.688719 18216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:54:03.689785 18216 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:54:03.691473 18216 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:54:03.691622 18216 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:54:03.691736 18216 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:54:03.691797 18216 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:54:03.733361 18216 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.199:42269
I20240502 01:54:03.733386 18346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.199:42269 every 8 connection(s)
I20240502 01:54:03.734779 18216 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 01:54:03.739989  9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu as pid 18216
I20240502 01:54:03.740170  9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240502 01:54:03.743535  9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
/tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.200:0
--local_ip_for_outbound_sockets=127.9.176.200
--webserver_interface=127.9.176.200
--webserver_port=0
--tserver_master_addrs=127.9.176.254:46743
--builtin_ntp_servers=127.9.176.212:41499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:54:03.750171 18347 heartbeater.cc:347] Connected to a master server at 127.9.176.254:46743
I20240502 01:54:03.750217 18081 heartbeater.cc:503] Master 127.9.176.254:46743 was elected leader, sending a full tablet report...
I20240502 01:54:03.750470 18347 heartbeater.cc:464] Registering TS with master...
I20240502 01:54:03.751415 18347 heartbeater.cc:511] Master 127.9.176.254:46743 requested a full tablet report, sending...
I20240502 01:54:03.753055 17356 ts_manager.cc:194] Registered new tserver with Master: 272b2179c1b4476090a89880df2a8baa (127.9.176.199:42269)
I20240502 01:54:03.753882 17356 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.176.199:41775
W20240502 01:54:04.125460 18350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:54:04.125804 18350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:54:04.125923 18350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:54:04.134035 18350 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:54:04.134186 18350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:54:04.134394 18350 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.200
I20240502 01:54:04.145421 18350 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:41499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.200:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.9.176.200
--webserver_port=0
--tserver_master_addrs=127.9.176.254:46743
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.200
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:54:04.146945 18350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:54:04.149124 18350 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:54:04.163117 18358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:04.163549 18355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:04.164216 18356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:54:04.164860 18350 server_base.cc:1034] running on GCE node
I20240502 01:54:04.165572 18350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:54:04.166666 18350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:54:04.167901 18350 hybrid_clock.cc:648] HybridClock initialized: now 1714614844167848 us; error 58 us; skew 500 ppm
I20240502 01:54:04.168294 18350 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:54:04.171097 18350 webserver.cc:460] Webserver started at http://127.9.176.200:39509/ using document root <none> and password file <none>
I20240502 01:54:04.172003 18350 fs_manager.cc:359] Metadata directory not provided
I20240502 01:54:04.172124 18350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:54:04.172448 18350 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:54:04.175438 18350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "94931f4b51c9493cb3d4264ce99f5e76"
format_stamp: "Formatted at 2024-05-02 01:54:04 on dist-test-slave-j3hh"
I20240502 01:54:04.176414 18350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "94931f4b51c9493cb3d4264ce99f5e76"
format_stamp: "Formatted at 2024-05-02 01:54:04 on dist-test-slave-j3hh"
I20240502 01:54:04.182490 18350 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.005s	sys 0.000s
I20240502 01:54:04.186494 18364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:54:04.188050 18350 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.001s
I20240502 01:54:04.188246 18350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "94931f4b51c9493cb3d4264ce99f5e76"
format_stamp: "Formatted at 2024-05-02 01:54:04 on dist-test-slave-j3hh"
I20240502 01:54:04.188475 18350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:54:04.211818 18350 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:54:04.212673 18350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:54:04.212942 18350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:54:04.213968 18350 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:54:04.215552 18350 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:54:04.215689 18350 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:54:04.215792 18350 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:54:04.215868 18350 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:54:04.247153 18213 heartbeater.cc:503] Master 127.9.176.254:46743 was elected leader, sending a full tablet report...
I20240502 01:54:04.251334 18350 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.200:37831
I20240502 01:54:04.251413 18479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.200:37831 every 8 connection(s)
I20240502 01:54:04.252822 18350 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 01:54:04.261559  9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu as pid 18350
I20240502 01:54:04.261765  9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240502 01:54:04.267496 18480 heartbeater.cc:347] Connected to a master server at 127.9.176.254:46743
I20240502 01:54:04.267900 18480 heartbeater.cc:464] Registering TS with master...
I20240502 01:54:04.268868 18480 heartbeater.cc:511] Master 127.9.176.254:46743 requested a full tablet report, sending...
I20240502 01:54:04.270568 17356 ts_manager.cc:194] Registered new tserver with Master: 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200:37831)
I20240502 01:54:04.271443 17356 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.176.200:40793
I20240502 01:54:04.279533  9923 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240502 01:54:04.297537  9923 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240502 01:54:04.297673  9923 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu pid 17951
I20240502 01:54:04.306612 18076 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:54:04.390332  9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu with pid 17951
I20240502 01:54:04.406955  9923 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu pid 18084
I20240502 01:54:04.416989 18208 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:54:04.499519  9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu with pid 18084
I20240502 01:54:04.515739  9923 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu pid 18216
I20240502 01:54:04.524431 18342 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:54:04.606618  9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu with pid 18216
I20240502 01:54:04.623598  9923 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu pid 18350
I20240502 01:54:04.632997 18475 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:54:04.716888  9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu with pid 18350
I20240502 01:54:08.490319 17356 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56698:
name: "rebalance_test_table_0"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 01:54:08.538300 17746 tablet_service.cc:1467] Processing CreateTablet for tablet 6da59c0626c64a5988e39a831c42c268 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c516e7439e734f02a0947c3680cf79e2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:08.540134 17747 tablet_service.cc:1467] Processing CreateTablet for tablet 62fd543d5c7c41f98926f2148ee61e7d (DEFAULT_TABLE table=rebalance_test_table_0 [id=c516e7439e734f02a0947c3680cf79e2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:08.544358 17880 tablet_service.cc:1467] Processing CreateTablet for tablet 7ce9d4281c1b46258493feb39e8d9ab2 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c516e7439e734f02a0947c3680cf79e2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:08.544358 17613 tablet_service.cc:1467] Processing CreateTablet for tablet 62fd543d5c7c41f98926f2148ee61e7d (DEFAULT_TABLE table=rebalance_test_table_0 [id=c516e7439e734f02a0947c3680cf79e2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:08.544950 17879 tablet_service.cc:1467] Processing CreateTablet for tablet 6da59c0626c64a5988e39a831c42c268 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c516e7439e734f02a0947c3680cf79e2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:08.546075 17613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62fd543d5c7c41f98926f2148ee61e7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:08.546744 17746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6da59c0626c64a5988e39a831c42c268. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:08.547402 17747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62fd543d5c7c41f98926f2148ee61e7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:08.546921 17479 tablet_service.cc:1467] Processing CreateTablet for tablet 62fd543d5c7c41f98926f2148ee61e7d (DEFAULT_TABLE table=rebalance_test_table_0 [id=c516e7439e734f02a0947c3680cf79e2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:08.547869 17612 tablet_service.cc:1467] Processing CreateTablet for tablet 7ce9d4281c1b46258493feb39e8d9ab2 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c516e7439e734f02a0947c3680cf79e2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:08.548410 17612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ce9d4281c1b46258493feb39e8d9ab2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:08.548704 17479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62fd543d5c7c41f98926f2148ee61e7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:08.549683 17477 tablet_service.cc:1467] Processing CreateTablet for tablet 6da59c0626c64a5988e39a831c42c268 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c516e7439e734f02a0947c3680cf79e2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:08.550287 17477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6da59c0626c64a5988e39a831c42c268. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:08.552273 17478 tablet_service.cc:1467] Processing CreateTablet for tablet 7ce9d4281c1b46258493feb39e8d9ab2 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c516e7439e734f02a0947c3680cf79e2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:08.553332 17880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ce9d4281c1b46258493feb39e8d9ab2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:08.554050 17478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ce9d4281c1b46258493feb39e8d9ab2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:08.556308 17879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6da59c0626c64a5988e39a831c42c268. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:08.573078 18511 tablet_bootstrap.cc:492] T 6da59c0626c64a5988e39a831c42c268 P 84d3a4b174c34e02aedab1e620731359: Bootstrap starting.
I20240502 01:54:08.578310 18513 tablet_bootstrap.cc:492] T 6da59c0626c64a5988e39a831c42c268 P 311f765f59754bdcb4eadc04e14bd4ea: Bootstrap starting.
I20240502 01:54:08.581373 18512 tablet_bootstrap.cc:492] T 7ce9d4281c1b46258493feb39e8d9ab2 P 1b50a031f2784ba498b3e52e16916205: Bootstrap starting.
I20240502 01:54:08.582620 18513 tablet_bootstrap.cc:654] T 6da59c0626c64a5988e39a831c42c268 P 311f765f59754bdcb4eadc04e14bd4ea: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:08.585423 18513 log.cc:824] T 6da59c0626c64a5988e39a831c42c268 P 311f765f59754bdcb4eadc04e14bd4ea: Log is configured to *not* fsync() on all Append() calls
I20240502 01:54:08.585646 18511 tablet_bootstrap.cc:654] T 6da59c0626c64a5988e39a831c42c268 P 84d3a4b174c34e02aedab1e620731359: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:08.586488 18512 tablet_bootstrap.cc:654] T 7ce9d4281c1b46258493feb39e8d9ab2 P 1b50a031f2784ba498b3e52e16916205: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:08.587343 18511 log.cc:824] T 6da59c0626c64a5988e39a831c42c268 P 84d3a4b174c34e02aedab1e620731359: Log is configured to *not* fsync() on all Append() calls
I20240502 01:54:08.587975 18512 log.cc:824] T 7ce9d4281c1b46258493feb39e8d9ab2 P 1b50a031f2784ba498b3e52e16916205: Log is configured to *not* fsync() on all Append() calls
I20240502 01:54:08.594887 18511 tablet_bootstrap.cc:492] T 6da59c0626c64a5988e39a831c42c268 P 84d3a4b174c34e02aedab1e620731359: No bootstrap required, opened a new log
I20240502 01:54:08.595376 18511 ts_tablet_manager.cc:1367] T 6da59c0626c64a5988e39a831c42c268 P 84d3a4b174c34e02aedab1e620731359: Time spent bootstrapping tablet: real 0.023s	user 0.011s	sys 0.000s
I20240502 01:54:08.601321 18518 tablet_bootstrap.cc:492] T 6da59c0626c64a5988e39a831c42c268 P da3584f429104d23a1a8899b35cc2138: Bootstrap starting.
I20240502 01:54:08.602839 18513 tablet_bootstrap.cc:492] T 6da59c0626c64a5988e39a831c42c268 P 311f765f59754bdcb4eadc04e14bd4ea: No bootstrap required, opened a new log
I20240502 01:54:08.603226 18513 ts_tablet_manager.cc:1367] T 6da59c0626c64a5988e39a831c42c268 P 311f765f59754bdcb4eadc04e14bd4ea: Time spent bootstrapping tablet: real 0.025s	user 0.013s	sys 0.007s
I20240502 01:54:08.605988 18512 tablet_bootstrap.cc:492] T 7ce9d4281c1b46258493feb39e8d9ab2 P 1b50a031f2784ba498b3e52e16916205: No bootstrap required, opened a new log
I20240502 01:54:08.606374 18512 ts_tablet_manager.cc:1367] T 7ce9d4281c1b46258493feb39e8d9ab2 P 1b50a031f2784ba498b3e52e16916205: Time spent bootstrapping tablet: real 0.025s	user 0.008s	sys 0.012s
I20240502 01:54:08.606626 18518 tablet_bootstrap.cc:654] T 6da59c0626c64a5988e39a831c42c268 P da3584f429104d23a1a8899b35cc2138: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:08.608155 18518 log.cc:824] T 6da59c0626c64a5988e39a831c42c268 P da3584f429104d23a1a8899b35cc2138: Log is configured to *not* fsync() on all Append() calls
I20240502 01:54:08.608064 18511 raft_consensus.cc:348] T 6da59c0626c64a5988e39a831c42c268 P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } }
I20240502 01:54:08.608665 18511 raft_consensus.cc:374] T 6da59c0626c64a5988e39a831c42c268 P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:08.608819 18511 raft_consensus.cc:729] T 6da59c0626c64a5988e39a831c42c268 P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84d3a4b174c34e02aedab1e620731359, State: Initialized, Role: FOLLOWER
I20240502 01:54:08.609686 18511 consensus_queue.cc:260] T 6da59c0626c64a5988e39a831c42c268 P 84d3a4b174c34e02aedab1e620731359 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } }
I20240502 01:54:08.614694 18513 raft_consensus.cc:348] T 6da59c0626c64a5988e39a831c42c268 P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } }
I20240502 01:54:08.615419 18513 raft_consensus.cc:374] T 6da59c0626c64a5988e39a831c42c268 P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:08.615589 18513 raft_consensus.cc:729] T 6da59c0626c64a5988e39a831c42c268 P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 311f765f59754bdcb4eadc04e14bd4ea, State: Initialized, Role: FOLLOWER
I20240502 01:54:08.616621 18513 consensus_queue.cc:260] T 6da59c0626c64a5988e39a831c42c268 P 311f765f59754bdcb4eadc04e14bd4ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } }
I20240502 01:54:08.618945 18512 raft_consensus.cc:348] T 7ce9d4281c1b46258493feb39e8d9ab2 P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } }
I20240502 01:54:08.619606 18512 raft_consensus.cc:374] T 7ce9d4281c1b46258493feb39e8d9ab2 P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:08.619767 18512 raft_consensus.cc:729] T 7ce9d4281c1b46258493feb39e8d9ab2 P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b50a031f2784ba498b3e52e16916205, State: Initialized, Role: FOLLOWER
I20240502 01:54:08.620348 18518 tablet_bootstrap.cc:492] T 6da59c0626c64a5988e39a831c42c268 P da3584f429104d23a1a8899b35cc2138: No bootstrap required, opened a new log
I20240502 01:54:08.620810 18518 ts_tablet_manager.cc:1367] T 6da59c0626c64a5988e39a831c42c268 P da3584f429104d23a1a8899b35cc2138: Time spent bootstrapping tablet: real 0.020s	user 0.010s	sys 0.005s
I20240502 01:54:08.620767 18512 consensus_queue.cc:260] T 7ce9d4281c1b46258493feb39e8d9ab2 P 1b50a031f2784ba498b3e52e16916205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } }
I20240502 01:54:08.624820 18511 ts_tablet_manager.cc:1398] T 6da59c0626c64a5988e39a831c42c268 P 84d3a4b174c34e02aedab1e620731359: Time spent starting tablet: real 0.029s	user 0.024s	sys 0.004s
I20240502 01:54:08.625332 18511 tablet_bootstrap.cc:492] T 62fd543d5c7c41f98926f2148ee61e7d P 84d3a4b174c34e02aedab1e620731359: Bootstrap starting.
I20240502 01:54:08.627866 18511 tablet_bootstrap.cc:654] T 62fd543d5c7c41f98926f2148ee61e7d P 84d3a4b174c34e02aedab1e620731359: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:08.631093 18512 ts_tablet_manager.cc:1398] T 7ce9d4281c1b46258493feb39e8d9ab2 P 1b50a031f2784ba498b3e52e16916205: Time spent starting tablet: real 0.025s	user 0.016s	sys 0.008s
I20240502 01:54:08.631913 18512 tablet_bootstrap.cc:492] T 62fd543d5c7c41f98926f2148ee61e7d P 1b50a031f2784ba498b3e52e16916205: Bootstrap starting.
I20240502 01:54:08.634481 18512 tablet_bootstrap.cc:654] T 62fd543d5c7c41f98926f2148ee61e7d P 1b50a031f2784ba498b3e52e16916205: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:08.637300 18511 tablet_bootstrap.cc:492] T 62fd543d5c7c41f98926f2148ee61e7d P 84d3a4b174c34e02aedab1e620731359: No bootstrap required, opened a new log
I20240502 01:54:08.637482 18512 tablet_bootstrap.cc:492] T 62fd543d5c7c41f98926f2148ee61e7d P 1b50a031f2784ba498b3e52e16916205: No bootstrap required, opened a new log
I20240502 01:54:08.637496 18511 ts_tablet_manager.cc:1367] T 62fd543d5c7c41f98926f2148ee61e7d P 84d3a4b174c34e02aedab1e620731359: Time spent bootstrapping tablet: real 0.012s	user 0.006s	sys 0.003s
I20240502 01:54:08.637729 18512 ts_tablet_manager.cc:1367] T 62fd543d5c7c41f98926f2148ee61e7d P 1b50a031f2784ba498b3e52e16916205: Time spent bootstrapping tablet: real 0.006s	user 0.002s	sys 0.003s
I20240502 01:54:08.638813 18511 raft_consensus.cc:348] T 62fd543d5c7c41f98926f2148ee61e7d P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } }
I20240502 01:54:08.639097 18511 raft_consensus.cc:374] T 62fd543d5c7c41f98926f2148ee61e7d P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:08.639204 18511 raft_consensus.cc:729] T 62fd543d5c7c41f98926f2148ee61e7d P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84d3a4b174c34e02aedab1e620731359, State: Initialized, Role: FOLLOWER
I20240502 01:54:08.639048 18512 raft_consensus.cc:348] T 62fd543d5c7c41f98926f2148ee61e7d P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } }
I20240502 01:54:08.639525 18512 raft_consensus.cc:374] T 62fd543d5c7c41f98926f2148ee61e7d P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:08.639452 18511 consensus_queue.cc:260] T 62fd543d5c7c41f98926f2148ee61e7d P 84d3a4b174c34e02aedab1e620731359 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } }
I20240502 01:54:08.639748 18512 raft_consensus.cc:729] T 62fd543d5c7c41f98926f2148ee61e7d P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b50a031f2784ba498b3e52e16916205, State: Initialized, Role: FOLLOWER
I20240502 01:54:08.640161 18512 consensus_queue.cc:260] T 62fd543d5c7c41f98926f2148ee61e7d P 1b50a031f2784ba498b3e52e16916205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } }
I20240502 01:54:08.640589 18511 ts_tablet_manager.cc:1398] T 62fd543d5c7c41f98926f2148ee61e7d P 84d3a4b174c34e02aedab1e620731359: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 01:54:08.640952 18512 ts_tablet_manager.cc:1398] T 62fd543d5c7c41f98926f2148ee61e7d P 1b50a031f2784ba498b3e52e16916205: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 01:54:08.643523 18513 ts_tablet_manager.cc:1398] T 6da59c0626c64a5988e39a831c42c268 P 311f765f59754bdcb4eadc04e14bd4ea: Time spent starting tablet: real 0.040s	user 0.029s	sys 0.012s
I20240502 01:54:08.644093 18513 tablet_bootstrap.cc:492] T 62fd543d5c7c41f98926f2148ee61e7d P 311f765f59754bdcb4eadc04e14bd4ea: Bootstrap starting.
I20240502 01:54:08.646078 18513 tablet_bootstrap.cc:654] T 62fd543d5c7c41f98926f2148ee61e7d P 311f765f59754bdcb4eadc04e14bd4ea: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:08.649885 18518 raft_consensus.cc:348] T 6da59c0626c64a5988e39a831c42c268 P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } }
I20240502 01:54:08.650298 18518 raft_consensus.cc:374] T 6da59c0626c64a5988e39a831c42c268 P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:08.650419 18518 raft_consensus.cc:729] T 6da59c0626c64a5988e39a831c42c268 P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da3584f429104d23a1a8899b35cc2138, State: Initialized, Role: FOLLOWER
I20240502 01:54:08.650676 18513 tablet_bootstrap.cc:492] T 62fd543d5c7c41f98926f2148ee61e7d P 311f765f59754bdcb4eadc04e14bd4ea: No bootstrap required, opened a new log
I20240502 01:54:08.650849 18513 ts_tablet_manager.cc:1367] T 62fd543d5c7c41f98926f2148ee61e7d P 311f765f59754bdcb4eadc04e14bd4ea: Time spent bootstrapping tablet: real 0.007s	user 0.004s	sys 0.002s
I20240502 01:54:08.651129 18518 consensus_queue.cc:260] T 6da59c0626c64a5988e39a831c42c268 P da3584f429104d23a1a8899b35cc2138 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } }
I20240502 01:54:08.651850 18513 raft_consensus.cc:348] T 62fd543d5c7c41f98926f2148ee61e7d P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } }
I20240502 01:54:08.652096 18513 raft_consensus.cc:374] T 62fd543d5c7c41f98926f2148ee61e7d P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:08.652233 18513 raft_consensus.cc:729] T 62fd543d5c7c41f98926f2148ee61e7d P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 311f765f59754bdcb4eadc04e14bd4ea, State: Initialized, Role: FOLLOWER
I20240502 01:54:08.652526 18513 consensus_queue.cc:260] T 62fd543d5c7c41f98926f2148ee61e7d P 311f765f59754bdcb4eadc04e14bd4ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } }
I20240502 01:54:08.653116 18513 ts_tablet_manager.cc:1398] T 62fd543d5c7c41f98926f2148ee61e7d P 311f765f59754bdcb4eadc04e14bd4ea: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 01:54:08.653389 18513 tablet_bootstrap.cc:492] T 7ce9d4281c1b46258493feb39e8d9ab2 P 311f765f59754bdcb4eadc04e14bd4ea: Bootstrap starting.
I20240502 01:54:08.654508 18518 ts_tablet_manager.cc:1398] T 6da59c0626c64a5988e39a831c42c268 P da3584f429104d23a1a8899b35cc2138: Time spent starting tablet: real 0.034s	user 0.009s	sys 0.004s
I20240502 01:54:08.655009 18518 tablet_bootstrap.cc:492] T 7ce9d4281c1b46258493feb39e8d9ab2 P da3584f429104d23a1a8899b35cc2138: Bootstrap starting.
I20240502 01:54:08.655262 18513 tablet_bootstrap.cc:654] T 7ce9d4281c1b46258493feb39e8d9ab2 P 311f765f59754bdcb4eadc04e14bd4ea: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:08.656986 18518 tablet_bootstrap.cc:654] T 7ce9d4281c1b46258493feb39e8d9ab2 P da3584f429104d23a1a8899b35cc2138: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:08.657685 18513 tablet_bootstrap.cc:492] T 7ce9d4281c1b46258493feb39e8d9ab2 P 311f765f59754bdcb4eadc04e14bd4ea: No bootstrap required, opened a new log
I20240502 01:54:08.657878 18513 ts_tablet_manager.cc:1367] T 7ce9d4281c1b46258493feb39e8d9ab2 P 311f765f59754bdcb4eadc04e14bd4ea: Time spent bootstrapping tablet: real 0.005s	user 0.000s	sys 0.003s
I20240502 01:54:08.658803 18513 raft_consensus.cc:348] T 7ce9d4281c1b46258493feb39e8d9ab2 P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } }
I20240502 01:54:08.659072 18513 raft_consensus.cc:374] T 7ce9d4281c1b46258493feb39e8d9ab2 P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:08.659157 18513 raft_consensus.cc:729] T 7ce9d4281c1b46258493feb39e8d9ab2 P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 311f765f59754bdcb4eadc04e14bd4ea, State: Initialized, Role: FOLLOWER
I20240502 01:54:08.659431 18513 consensus_queue.cc:260] T 7ce9d4281c1b46258493feb39e8d9ab2 P 311f765f59754bdcb4eadc04e14bd4ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } }
I20240502 01:54:08.659626 18518 tablet_bootstrap.cc:492] T 7ce9d4281c1b46258493feb39e8d9ab2 P da3584f429104d23a1a8899b35cc2138: No bootstrap required, opened a new log
I20240502 01:54:08.659785 18518 ts_tablet_manager.cc:1367] T 7ce9d4281c1b46258493feb39e8d9ab2 P da3584f429104d23a1a8899b35cc2138: Time spent bootstrapping tablet: real 0.005s	user 0.003s	sys 0.001s
I20240502 01:54:08.660194 18513 ts_tablet_manager.cc:1398] T 7ce9d4281c1b46258493feb39e8d9ab2 P 311f765f59754bdcb4eadc04e14bd4ea: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.000s
I20240502 01:54:08.660887 18518 raft_consensus.cc:348] T 7ce9d4281c1b46258493feb39e8d9ab2 P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } }
I20240502 01:54:08.661069 18518 raft_consensus.cc:374] T 7ce9d4281c1b46258493feb39e8d9ab2 P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:08.661181 18518 raft_consensus.cc:729] T 7ce9d4281c1b46258493feb39e8d9ab2 P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da3584f429104d23a1a8899b35cc2138, State: Initialized, Role: FOLLOWER
I20240502 01:54:08.661433 18518 consensus_queue.cc:260] T 7ce9d4281c1b46258493feb39e8d9ab2 P da3584f429104d23a1a8899b35cc2138 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } }
I20240502 01:54:08.661972 18518 ts_tablet_manager.cc:1398] T 7ce9d4281c1b46258493feb39e8d9ab2 P da3584f429104d23a1a8899b35cc2138: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
W20240502 01:54:08.707542 17815 tablet.cc:2369] T 62fd543d5c7c41f98926f2148ee61e7d P 84d3a4b174c34e02aedab1e620731359: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 01:54:08.712288 17682 tablet.cc:2369] T 62fd543d5c7c41f98926f2148ee61e7d P 1b50a031f2784ba498b3e52e16916205: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 01:54:08.724155 17546 tablet.cc:2369] T 62fd543d5c7c41f98926f2148ee61e7d P 311f765f59754bdcb4eadc04e14bd4ea: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 01:54:08.724710 17948 tablet.cc:2369] T 6da59c0626c64a5988e39a831c42c268 P da3584f429104d23a1a8899b35cc2138: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:54:08.802568 18521 raft_consensus.cc:482] T 62fd543d5c7c41f98926f2148ee61e7d P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:08.802863 18521 raft_consensus.cc:504] T 62fd543d5c7c41f98926f2148ee61e7d P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } }
I20240502 01:54:08.804795 18521 leader_election.cc:288] T 62fd543d5c7c41f98926f2148ee61e7d P 311f765f59754bdcb4eadc04e14bd4ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84d3a4b174c34e02aedab1e620731359 (127.9.176.195:37357), 1b50a031f2784ba498b3e52e16916205 (127.9.176.194:45617)
I20240502 01:54:08.811806 17767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62fd543d5c7c41f98926f2148ee61e7d" candidate_uuid: "311f765f59754bdcb4eadc04e14bd4ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84d3a4b174c34e02aedab1e620731359" is_pre_election: true
I20240502 01:54:08.812345 17767 raft_consensus.cc:2454] T 62fd543d5c7c41f98926f2148ee61e7d P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 311f765f59754bdcb4eadc04e14bd4ea in term 0.
I20240502 01:54:08.812309 17634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62fd543d5c7c41f98926f2148ee61e7d" candidate_uuid: "311f765f59754bdcb4eadc04e14bd4ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b50a031f2784ba498b3e52e16916205" is_pre_election: true
I20240502 01:54:08.812819 17634 raft_consensus.cc:2454] T 62fd543d5c7c41f98926f2148ee61e7d P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 311f765f59754bdcb4eadc04e14bd4ea in term 0.
I20240502 01:54:08.813016 17435 leader_election.cc:302] T 62fd543d5c7c41f98926f2148ee61e7d P 311f765f59754bdcb4eadc04e14bd4ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 311f765f59754bdcb4eadc04e14bd4ea, 84d3a4b174c34e02aedab1e620731359; no voters: 
I20240502 01:54:08.813434 18521 raft_consensus.cc:2781] T 62fd543d5c7c41f98926f2148ee61e7d P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:08.813578 18521 raft_consensus.cc:482] T 62fd543d5c7c41f98926f2148ee61e7d P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:08.813697 18521 raft_consensus.cc:3037] T 62fd543d5c7c41f98926f2148ee61e7d P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:08.815896 18521 raft_consensus.cc:504] T 62fd543d5c7c41f98926f2148ee61e7d P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } }
I20240502 01:54:08.816500 18521 leader_election.cc:288] T 62fd543d5c7c41f98926f2148ee61e7d P 311f765f59754bdcb4eadc04e14bd4ea [CANDIDATE]: Term 1 election: Requested vote from peers 84d3a4b174c34e02aedab1e620731359 (127.9.176.195:37357), 1b50a031f2784ba498b3e52e16916205 (127.9.176.194:45617)
I20240502 01:54:08.816965 17767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62fd543d5c7c41f98926f2148ee61e7d" candidate_uuid: "311f765f59754bdcb4eadc04e14bd4ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84d3a4b174c34e02aedab1e620731359"
I20240502 01:54:08.817070 17634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62fd543d5c7c41f98926f2148ee61e7d" candidate_uuid: "311f765f59754bdcb4eadc04e14bd4ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b50a031f2784ba498b3e52e16916205"
I20240502 01:54:08.817147 17767 raft_consensus.cc:3037] T 62fd543d5c7c41f98926f2148ee61e7d P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:08.817282 17634 raft_consensus.cc:3037] T 62fd543d5c7c41f98926f2148ee61e7d P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:08.819217 17767 raft_consensus.cc:2454] T 62fd543d5c7c41f98926f2148ee61e7d P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 311f765f59754bdcb4eadc04e14bd4ea in term 1.
I20240502 01:54:08.819358 17634 raft_consensus.cc:2454] T 62fd543d5c7c41f98926f2148ee61e7d P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 311f765f59754bdcb4eadc04e14bd4ea in term 1.
I20240502 01:54:08.819691 17435 leader_election.cc:302] T 62fd543d5c7c41f98926f2148ee61e7d P 311f765f59754bdcb4eadc04e14bd4ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 311f765f59754bdcb4eadc04e14bd4ea, 84d3a4b174c34e02aedab1e620731359; no voters: 
I20240502 01:54:08.820008 18521 raft_consensus.cc:2781] T 62fd543d5c7c41f98926f2148ee61e7d P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:08.820441 18521 raft_consensus.cc:686] T 62fd543d5c7c41f98926f2148ee61e7d P 311f765f59754bdcb4eadc04e14bd4ea [term 1 LEADER]: Becoming Leader. State: Replica: 311f765f59754bdcb4eadc04e14bd4ea, State: Running, Role: LEADER
I20240502 01:54:08.820999 18521 consensus_queue.cc:237] T 62fd543d5c7c41f98926f2148ee61e7d P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } }
I20240502 01:54:08.825810 17355 catalog_manager.cc:5484] T 62fd543d5c7c41f98926f2148ee61e7d P 311f765f59754bdcb4eadc04e14bd4ea reported cstate change: term changed from 0 to 1, leader changed from <none> to 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193). New cstate: current_term: 1 leader_uuid: "311f765f59754bdcb4eadc04e14bd4ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:08.872684 18521 raft_consensus.cc:482] T 7ce9d4281c1b46258493feb39e8d9ab2 P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:08.872983 18521 raft_consensus.cc:504] T 7ce9d4281c1b46258493feb39e8d9ab2 P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } }
I20240502 01:54:08.873791 18521 leader_election.cc:288] T 7ce9d4281c1b46258493feb39e8d9ab2 P 311f765f59754bdcb4eadc04e14bd4ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers da3584f429104d23a1a8899b35cc2138 (127.9.176.196:46381), 1b50a031f2784ba498b3e52e16916205 (127.9.176.194:45617)
I20240502 01:54:08.874517 17634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ce9d4281c1b46258493feb39e8d9ab2" candidate_uuid: "311f765f59754bdcb4eadc04e14bd4ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b50a031f2784ba498b3e52e16916205" is_pre_election: true
I20240502 01:54:08.874795 17634 raft_consensus.cc:2454] T 7ce9d4281c1b46258493feb39e8d9ab2 P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 311f765f59754bdcb4eadc04e14bd4ea in term 0.
I20240502 01:54:08.875299 17435 leader_election.cc:302] T 7ce9d4281c1b46258493feb39e8d9ab2 P 311f765f59754bdcb4eadc04e14bd4ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b50a031f2784ba498b3e52e16916205, 311f765f59754bdcb4eadc04e14bd4ea; no voters: 
I20240502 01:54:08.875684 18521 raft_consensus.cc:2781] T 7ce9d4281c1b46258493feb39e8d9ab2 P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:08.875810 18521 raft_consensus.cc:482] T 7ce9d4281c1b46258493feb39e8d9ab2 P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:08.875936 18521 raft_consensus.cc:3037] T 7ce9d4281c1b46258493feb39e8d9ab2 P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:08.878696 18521 raft_consensus.cc:504] T 7ce9d4281c1b46258493feb39e8d9ab2 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } }
I20240502 01:54:08.879556 18521 leader_election.cc:288] T 7ce9d4281c1b46258493feb39e8d9ab2 P 311f765f59754bdcb4eadc04e14bd4ea [CANDIDATE]: Term 1 election: Requested vote from peers da3584f429104d23a1a8899b35cc2138 (127.9.176.196:46381), 1b50a031f2784ba498b3e52e16916205 (127.9.176.194:45617)
I20240502 01:54:08.880278 17634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ce9d4281c1b46258493feb39e8d9ab2" candidate_uuid: "311f765f59754bdcb4eadc04e14bd4ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b50a031f2784ba498b3e52e16916205"
I20240502 01:54:08.880584 17634 raft_consensus.cc:3037] T 7ce9d4281c1b46258493feb39e8d9ab2 P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:08.881484 17901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ce9d4281c1b46258493feb39e8d9ab2" candidate_uuid: "311f765f59754bdcb4eadc04e14bd4ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da3584f429104d23a1a8899b35cc2138" is_pre_election: true
I20240502 01:54:08.881536 17900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ce9d4281c1b46258493feb39e8d9ab2" candidate_uuid: "311f765f59754bdcb4eadc04e14bd4ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da3584f429104d23a1a8899b35cc2138"
I20240502 01:54:08.882031 17901 raft_consensus.cc:2454] T 7ce9d4281c1b46258493feb39e8d9ab2 P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 311f765f59754bdcb4eadc04e14bd4ea in term 0.
I20240502 01:54:08.883200 17634 raft_consensus.cc:2454] T 7ce9d4281c1b46258493feb39e8d9ab2 P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 311f765f59754bdcb4eadc04e14bd4ea in term 1.
I20240502 01:54:08.883711 17435 leader_election.cc:302] T 7ce9d4281c1b46258493feb39e8d9ab2 P 311f765f59754bdcb4eadc04e14bd4ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b50a031f2784ba498b3e52e16916205, 311f765f59754bdcb4eadc04e14bd4ea; no voters: da3584f429104d23a1a8899b35cc2138
I20240502 01:54:08.884136 18521 raft_consensus.cc:2781] T 7ce9d4281c1b46258493feb39e8d9ab2 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:08.884344 18521 raft_consensus.cc:686] T 7ce9d4281c1b46258493feb39e8d9ab2 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 LEADER]: Becoming Leader. State: Replica: 311f765f59754bdcb4eadc04e14bd4ea, State: Running, Role: LEADER
I20240502 01:54:08.884634 18521 consensus_queue.cc:237] T 7ce9d4281c1b46258493feb39e8d9ab2 P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } }
I20240502 01:54:08.887746 17355 catalog_manager.cc:5484] T 7ce9d4281c1b46258493feb39e8d9ab2 P 311f765f59754bdcb4eadc04e14bd4ea reported cstate change: term changed from 0 to 1, leader changed from <none> to 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193). New cstate: current_term: 1 leader_uuid: "311f765f59754bdcb4eadc04e14bd4ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:08.997763 18520 raft_consensus.cc:482] T 6da59c0626c64a5988e39a831c42c268 P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:08.998023 18520 raft_consensus.cc:504] T 6da59c0626c64a5988e39a831c42c268 P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } }
I20240502 01:54:08.999783 18520 leader_election.cc:288] T 6da59c0626c64a5988e39a831c42c268 P 84d3a4b174c34e02aedab1e620731359 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193:35397), da3584f429104d23a1a8899b35cc2138 (127.9.176.196:46381)
I20240502 01:54:09.007041 17900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6da59c0626c64a5988e39a831c42c268" candidate_uuid: "84d3a4b174c34e02aedab1e620731359" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da3584f429104d23a1a8899b35cc2138" is_pre_election: true
I20240502 01:54:09.007352 17900 raft_consensus.cc:2454] T 6da59c0626c64a5988e39a831c42c268 P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84d3a4b174c34e02aedab1e620731359 in term 0.
I20240502 01:54:09.007601 17500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6da59c0626c64a5988e39a831c42c268" candidate_uuid: "84d3a4b174c34e02aedab1e620731359" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "311f765f59754bdcb4eadc04e14bd4ea" is_pre_election: true
I20240502 01:54:09.007896 17700 leader_election.cc:302] T 6da59c0626c64a5988e39a831c42c268 P 84d3a4b174c34e02aedab1e620731359 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84d3a4b174c34e02aedab1e620731359, da3584f429104d23a1a8899b35cc2138; no voters: 
I20240502 01:54:09.008083 17500 raft_consensus.cc:2454] T 6da59c0626c64a5988e39a831c42c268 P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84d3a4b174c34e02aedab1e620731359 in term 0.
I20240502 01:54:09.008339 18520 raft_consensus.cc:2781] T 6da59c0626c64a5988e39a831c42c268 P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:09.008469 18520 raft_consensus.cc:482] T 6da59c0626c64a5988e39a831c42c268 P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:09.008594 18520 raft_consensus.cc:3037] T 6da59c0626c64a5988e39a831c42c268 P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:09.010617 18520 raft_consensus.cc:504] T 6da59c0626c64a5988e39a831c42c268 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } }
I20240502 01:54:09.011224 18520 leader_election.cc:288] T 6da59c0626c64a5988e39a831c42c268 P 84d3a4b174c34e02aedab1e620731359 [CANDIDATE]: Term 1 election: Requested vote from peers 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193:35397), da3584f429104d23a1a8899b35cc2138 (127.9.176.196:46381)
I20240502 01:54:09.011788 17900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6da59c0626c64a5988e39a831c42c268" candidate_uuid: "84d3a4b174c34e02aedab1e620731359" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da3584f429104d23a1a8899b35cc2138"
I20240502 01:54:09.011790 17500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6da59c0626c64a5988e39a831c42c268" candidate_uuid: "84d3a4b174c34e02aedab1e620731359" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "311f765f59754bdcb4eadc04e14bd4ea"
I20240502 01:54:09.011983 17500 raft_consensus.cc:3037] T 6da59c0626c64a5988e39a831c42c268 P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:09.011998 17900 raft_consensus.cc:3037] T 6da59c0626c64a5988e39a831c42c268 P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:09.014026 17500 raft_consensus.cc:2454] T 6da59c0626c64a5988e39a831c42c268 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84d3a4b174c34e02aedab1e620731359 in term 1.
I20240502 01:54:09.014078 17900 raft_consensus.cc:2454] T 6da59c0626c64a5988e39a831c42c268 P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84d3a4b174c34e02aedab1e620731359 in term 1.
I20240502 01:54:09.014487 17703 leader_election.cc:302] T 6da59c0626c64a5988e39a831c42c268 P 84d3a4b174c34e02aedab1e620731359 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 311f765f59754bdcb4eadc04e14bd4ea, 84d3a4b174c34e02aedab1e620731359; no voters: 
I20240502 01:54:09.014792 18520 raft_consensus.cc:2781] T 6da59c0626c64a5988e39a831c42c268 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:09.015237 18520 raft_consensus.cc:686] T 6da59c0626c64a5988e39a831c42c268 P 84d3a4b174c34e02aedab1e620731359 [term 1 LEADER]: Becoming Leader. State: Replica: 84d3a4b174c34e02aedab1e620731359, State: Running, Role: LEADER
I20240502 01:54:09.015784 18520 consensus_queue.cc:237] T 6da59c0626c64a5988e39a831c42c268 P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } }
I20240502 01:54:09.020203 17355 catalog_manager.cc:5484] T 6da59c0626c64a5988e39a831c42c268 P 84d3a4b174c34e02aedab1e620731359 reported cstate change: term changed from 0 to 1, leader changed from <none> to 84d3a4b174c34e02aedab1e620731359 (127.9.176.195). New cstate: current_term: 1 leader_uuid: "84d3a4b174c34e02aedab1e620731359" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:09.220115 18521 consensus_queue.cc:1035] T 62fd543d5c7c41f98926f2148ee61e7d P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:09.234195 18521 consensus_queue.cc:1035] T 62fd543d5c7c41f98926f2148ee61e7d P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:09.406533 18520 consensus_queue.cc:1035] T 6da59c0626c64a5988e39a831c42c268 P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:09.415192 18520 consensus_queue.cc:1035] T 6da59c0626c64a5988e39a831c42c268 P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:54:09.472681 18538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:54:09.473038 18538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:54:09.481079 18538 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 01:54:09.484159 17900 raft_consensus.cc:3037] T 7ce9d4281c1b46258493feb39e8d9ab2 P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:09.487169 18521 consensus_queue.cc:1035] T 7ce9d4281c1b46258493feb39e8d9ab2 P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:09.494287 18521 consensus_queue.cc:1035] T 7ce9d4281c1b46258493feb39e8d9ab2 P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_0'

INSERT report
    rows total: 2000
    time total: 254.962 ms
  time per row: 0.127481 ms
-----------------------------------------------------
Suppressions used:
  count      bytes template
      2        596 libcrypto.so.1.1
-----------------------------------------------------

I20240502 01:54:10.828155 17353 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56698:
name: "rebalance_test_table_1"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 01:54:10.839700 17612 tablet_service.cc:1467] Processing CreateTablet for tablet 2045e08a200148c69326351f38fd484d (DEFAULT_TABLE table=rebalance_test_table_1 [id=952912d7ec8d493bbb8985fffe9aa918]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:10.839928 17613 tablet_service.cc:1467] Processing CreateTablet for tablet 768e7cbf63a94d3a815ac77a4d0acf41 (DEFAULT_TABLE table=rebalance_test_table_1 [id=952912d7ec8d493bbb8985fffe9aa918]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:10.839870 17747 tablet_service.cc:1467] Processing CreateTablet for tablet 2045e08a200148c69326351f38fd484d (DEFAULT_TABLE table=rebalance_test_table_1 [id=952912d7ec8d493bbb8985fffe9aa918]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:10.839900 17746 tablet_service.cc:1467] Processing CreateTablet for tablet 768e7cbf63a94d3a815ac77a4d0acf41 (DEFAULT_TABLE table=rebalance_test_table_1 [id=952912d7ec8d493bbb8985fffe9aa918]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:10.840329 17612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2045e08a200148c69326351f38fd484d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:10.840508 17746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 768e7cbf63a94d3a815ac77a4d0acf41. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:10.840781 17747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2045e08a200148c69326351f38fd484d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:10.841046 17613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 768e7cbf63a94d3a815ac77a4d0acf41. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:10.844166 17879 tablet_service.cc:1467] Processing CreateTablet for tablet 2045e08a200148c69326351f38fd484d (DEFAULT_TABLE table=rebalance_test_table_1 [id=952912d7ec8d493bbb8985fffe9aa918]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:10.844714 17879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2045e08a200148c69326351f38fd484d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:10.844848 17611 tablet_service.cc:1467] Processing CreateTablet for tablet 37eda67598fe491bbb29aa62b2c4bec7 (DEFAULT_TABLE table=rebalance_test_table_1 [id=952912d7ec8d493bbb8985fffe9aa918]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:10.845445 17611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37eda67598fe491bbb29aa62b2c4bec7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:10.848016 17880 tablet_service.cc:1467] Processing CreateTablet for tablet 768e7cbf63a94d3a815ac77a4d0acf41 (DEFAULT_TABLE table=rebalance_test_table_1 [id=952912d7ec8d493bbb8985fffe9aa918]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:10.848479 17880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 768e7cbf63a94d3a815ac77a4d0acf41. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:10.850307 18606 tablet_bootstrap.cc:492] T 2045e08a200148c69326351f38fd484d P 84d3a4b174c34e02aedab1e620731359: Bootstrap starting.
I20240502 01:54:10.852237 18606 tablet_bootstrap.cc:654] T 2045e08a200148c69326351f38fd484d P 84d3a4b174c34e02aedab1e620731359: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:10.852433 17745 tablet_service.cc:1467] Processing CreateTablet for tablet 37eda67598fe491bbb29aa62b2c4bec7 (DEFAULT_TABLE table=rebalance_test_table_1 [id=952912d7ec8d493bbb8985fffe9aa918]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:10.852876 17745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37eda67598fe491bbb29aa62b2c4bec7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:10.854483 18606 tablet_bootstrap.cc:492] T 2045e08a200148c69326351f38fd484d P 84d3a4b174c34e02aedab1e620731359: No bootstrap required, opened a new log
I20240502 01:54:10.854665 18606 ts_tablet_manager.cc:1367] T 2045e08a200148c69326351f38fd484d P 84d3a4b174c34e02aedab1e620731359: Time spent bootstrapping tablet: real 0.005s	user 0.003s	sys 0.000s
I20240502 01:54:10.855639 18606 raft_consensus.cc:348] T 2045e08a200148c69326351f38fd484d P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } }
I20240502 01:54:10.855830 18606 raft_consensus.cc:374] T 2045e08a200148c69326351f38fd484d P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:10.855918 18606 raft_consensus.cc:729] T 2045e08a200148c69326351f38fd484d P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84d3a4b174c34e02aedab1e620731359, State: Initialized, Role: FOLLOWER
I20240502 01:54:10.856145 18606 consensus_queue.cc:260] T 2045e08a200148c69326351f38fd484d P 84d3a4b174c34e02aedab1e620731359 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } }
I20240502 01:54:10.856598 18606 ts_tablet_manager.cc:1398] T 2045e08a200148c69326351f38fd484d P 84d3a4b174c34e02aedab1e620731359: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240502 01:54:10.856892 18606 tablet_bootstrap.cc:492] T 768e7cbf63a94d3a815ac77a4d0acf41 P 84d3a4b174c34e02aedab1e620731359: Bootstrap starting.
I20240502 01:54:10.856983 17478 tablet_service.cc:1467] Processing CreateTablet for tablet 37eda67598fe491bbb29aa62b2c4bec7 (DEFAULT_TABLE table=rebalance_test_table_1 [id=952912d7ec8d493bbb8985fffe9aa918]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:10.857570 17478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37eda67598fe491bbb29aa62b2c4bec7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:10.858500 18606 tablet_bootstrap.cc:654] T 768e7cbf63a94d3a815ac77a4d0acf41 P 84d3a4b174c34e02aedab1e620731359: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:10.864295 18607 tablet_bootstrap.cc:492] T 37eda67598fe491bbb29aa62b2c4bec7 P 1b50a031f2784ba498b3e52e16916205: Bootstrap starting.
I20240502 01:54:10.864385 18606 tablet_bootstrap.cc:492] T 768e7cbf63a94d3a815ac77a4d0acf41 P 84d3a4b174c34e02aedab1e620731359: No bootstrap required, opened a new log
I20240502 01:54:10.864564 18606 ts_tablet_manager.cc:1367] T 768e7cbf63a94d3a815ac77a4d0acf41 P 84d3a4b174c34e02aedab1e620731359: Time spent bootstrapping tablet: real 0.008s	user 0.001s	sys 0.002s
I20240502 01:54:10.865407 18606 raft_consensus.cc:348] T 768e7cbf63a94d3a815ac77a4d0acf41 P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } }
I20240502 01:54:10.865679 18606 raft_consensus.cc:374] T 768e7cbf63a94d3a815ac77a4d0acf41 P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:10.865768 18606 raft_consensus.cc:729] T 768e7cbf63a94d3a815ac77a4d0acf41 P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84d3a4b174c34e02aedab1e620731359, State: Initialized, Role: FOLLOWER
I20240502 01:54:10.866205 18606 consensus_queue.cc:260] T 768e7cbf63a94d3a815ac77a4d0acf41 P 84d3a4b174c34e02aedab1e620731359 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } }
I20240502 01:54:10.867823 18607 tablet_bootstrap.cc:654] T 37eda67598fe491bbb29aa62b2c4bec7 P 1b50a031f2784ba498b3e52e16916205: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:10.868815 18606 ts_tablet_manager.cc:1398] T 768e7cbf63a94d3a815ac77a4d0acf41 P 84d3a4b174c34e02aedab1e620731359: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.002s
I20240502 01:54:10.879400 18613 tablet_bootstrap.cc:492] T 37eda67598fe491bbb29aa62b2c4bec7 P 311f765f59754bdcb4eadc04e14bd4ea: Bootstrap starting.
I20240502 01:54:10.879976 18607 tablet_bootstrap.cc:492] T 37eda67598fe491bbb29aa62b2c4bec7 P 1b50a031f2784ba498b3e52e16916205: No bootstrap required, opened a new log
I20240502 01:54:10.880247 18607 ts_tablet_manager.cc:1367] T 37eda67598fe491bbb29aa62b2c4bec7 P 1b50a031f2784ba498b3e52e16916205: Time spent bootstrapping tablet: real 0.016s	user 0.002s	sys 0.003s
I20240502 01:54:10.881217 18607 raft_consensus.cc:348] T 37eda67598fe491bbb29aa62b2c4bec7 P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } }
I20240502 01:54:10.881536 18613 tablet_bootstrap.cc:654] T 37eda67598fe491bbb29aa62b2c4bec7 P 311f765f59754bdcb4eadc04e14bd4ea: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:10.882987 18607 raft_consensus.cc:374] T 37eda67598fe491bbb29aa62b2c4bec7 P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:10.883253 18607 raft_consensus.cc:729] T 37eda67598fe491bbb29aa62b2c4bec7 P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b50a031f2784ba498b3e52e16916205, State: Initialized, Role: FOLLOWER
I20240502 01:54:10.884024 18606 tablet_bootstrap.cc:492] T 37eda67598fe491bbb29aa62b2c4bec7 P 84d3a4b174c34e02aedab1e620731359: Bootstrap starting.
I20240502 01:54:10.883800 18607 consensus_queue.cc:260] T 37eda67598fe491bbb29aa62b2c4bec7 P 1b50a031f2784ba498b3e52e16916205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } }
I20240502 01:54:10.886566 18606 tablet_bootstrap.cc:654] T 37eda67598fe491bbb29aa62b2c4bec7 P 84d3a4b174c34e02aedab1e620731359: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:10.890827 18612 tablet_bootstrap.cc:492] T 2045e08a200148c69326351f38fd484d P da3584f429104d23a1a8899b35cc2138: Bootstrap starting.
I20240502 01:54:10.891170 18606 tablet_bootstrap.cc:492] T 37eda67598fe491bbb29aa62b2c4bec7 P 84d3a4b174c34e02aedab1e620731359: No bootstrap required, opened a new log
I20240502 01:54:10.891335 18606 ts_tablet_manager.cc:1367] T 37eda67598fe491bbb29aa62b2c4bec7 P 84d3a4b174c34e02aedab1e620731359: Time spent bootstrapping tablet: real 0.007s	user 0.002s	sys 0.002s
I20240502 01:54:10.892211 18606 raft_consensus.cc:348] T 37eda67598fe491bbb29aa62b2c4bec7 P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } }
I20240502 01:54:10.892444 18606 raft_consensus.cc:374] T 37eda67598fe491bbb29aa62b2c4bec7 P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:10.892560 18606 raft_consensus.cc:729] T 37eda67598fe491bbb29aa62b2c4bec7 P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84d3a4b174c34e02aedab1e620731359, State: Initialized, Role: FOLLOWER
I20240502 01:54:10.892560 18607 ts_tablet_manager.cc:1398] T 37eda67598fe491bbb29aa62b2c4bec7 P 1b50a031f2784ba498b3e52e16916205: Time spent starting tablet: real 0.012s	user 0.002s	sys 0.007s
I20240502 01:54:10.892817 18606 consensus_queue.cc:260] T 37eda67598fe491bbb29aa62b2c4bec7 P 84d3a4b174c34e02aedab1e620731359 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } }
I20240502 01:54:10.893201 18607 tablet_bootstrap.cc:492] T 768e7cbf63a94d3a815ac77a4d0acf41 P 1b50a031f2784ba498b3e52e16916205: Bootstrap starting.
I20240502 01:54:10.893438 18606 ts_tablet_manager.cc:1398] T 37eda67598fe491bbb29aa62b2c4bec7 P 84d3a4b174c34e02aedab1e620731359: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 01:54:10.895419 18607 tablet_bootstrap.cc:654] T 768e7cbf63a94d3a815ac77a4d0acf41 P 1b50a031f2784ba498b3e52e16916205: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:10.900687 18613 tablet_bootstrap.cc:492] T 37eda67598fe491bbb29aa62b2c4bec7 P 311f765f59754bdcb4eadc04e14bd4ea: No bootstrap required, opened a new log
I20240502 01:54:10.900884 18613 ts_tablet_manager.cc:1367] T 37eda67598fe491bbb29aa62b2c4bec7 P 311f765f59754bdcb4eadc04e14bd4ea: Time spent bootstrapping tablet: real 0.022s	user 0.006s	sys 0.013s
I20240502 01:54:10.900950 18607 tablet_bootstrap.cc:492] T 768e7cbf63a94d3a815ac77a4d0acf41 P 1b50a031f2784ba498b3e52e16916205: No bootstrap required, opened a new log
I20240502 01:54:10.900982 18612 tablet_bootstrap.cc:654] T 2045e08a200148c69326351f38fd484d P da3584f429104d23a1a8899b35cc2138: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:10.901156 18607 ts_tablet_manager.cc:1367] T 768e7cbf63a94d3a815ac77a4d0acf41 P 1b50a031f2784ba498b3e52e16916205: Time spent bootstrapping tablet: real 0.008s	user 0.007s	sys 0.000s
I20240502 01:54:10.901629 18613 raft_consensus.cc:348] T 37eda67598fe491bbb29aa62b2c4bec7 P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } }
I20240502 01:54:10.901816 18613 raft_consensus.cc:374] T 37eda67598fe491bbb29aa62b2c4bec7 P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:10.901926 18613 raft_consensus.cc:729] T 37eda67598fe491bbb29aa62b2c4bec7 P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 311f765f59754bdcb4eadc04e14bd4ea, State: Initialized, Role: FOLLOWER
I20240502 01:54:10.902065 18607 raft_consensus.cc:348] T 768e7cbf63a94d3a815ac77a4d0acf41 P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } }
I20240502 01:54:10.902314 18607 raft_consensus.cc:374] T 768e7cbf63a94d3a815ac77a4d0acf41 P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:10.902253 18613 consensus_queue.cc:260] T 37eda67598fe491bbb29aa62b2c4bec7 P 311f765f59754bdcb4eadc04e14bd4ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } }
I20240502 01:54:10.902408 18607 raft_consensus.cc:729] T 768e7cbf63a94d3a815ac77a4d0acf41 P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b50a031f2784ba498b3e52e16916205, State: Initialized, Role: FOLLOWER
I20240502 01:54:10.902702 18607 consensus_queue.cc:260] T 768e7cbf63a94d3a815ac77a4d0acf41 P 1b50a031f2784ba498b3e52e16916205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } }
I20240502 01:54:10.902859 18613 ts_tablet_manager.cc:1398] T 37eda67598fe491bbb29aa62b2c4bec7 P 311f765f59754bdcb4eadc04e14bd4ea: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240502 01:54:10.903281 18607 ts_tablet_manager.cc:1398] T 768e7cbf63a94d3a815ac77a4d0acf41 P 1b50a031f2784ba498b3e52e16916205: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 01:54:10.903703 18607 tablet_bootstrap.cc:492] T 2045e08a200148c69326351f38fd484d P 1b50a031f2784ba498b3e52e16916205: Bootstrap starting.
I20240502 01:54:10.904131 18612 tablet_bootstrap.cc:492] T 2045e08a200148c69326351f38fd484d P da3584f429104d23a1a8899b35cc2138: No bootstrap required, opened a new log
I20240502 01:54:10.904335 18612 ts_tablet_manager.cc:1367] T 2045e08a200148c69326351f38fd484d P da3584f429104d23a1a8899b35cc2138: Time spent bootstrapping tablet: real 0.014s	user 0.004s	sys 0.000s
I20240502 01:54:10.905290 18612 raft_consensus.cc:348] T 2045e08a200148c69326351f38fd484d P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } }
I20240502 01:54:10.905441 18607 tablet_bootstrap.cc:654] T 2045e08a200148c69326351f38fd484d P 1b50a031f2784ba498b3e52e16916205: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:10.905542 18612 raft_consensus.cc:374] T 2045e08a200148c69326351f38fd484d P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:10.905663 18612 raft_consensus.cc:729] T 2045e08a200148c69326351f38fd484d P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da3584f429104d23a1a8899b35cc2138, State: Initialized, Role: FOLLOWER
I20240502 01:54:10.906031 18612 consensus_queue.cc:260] T 2045e08a200148c69326351f38fd484d P da3584f429104d23a1a8899b35cc2138 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } }
I20240502 01:54:10.909299 18612 ts_tablet_manager.cc:1398] T 2045e08a200148c69326351f38fd484d P da3584f429104d23a1a8899b35cc2138: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.000s
I20240502 01:54:10.909530 18607 tablet_bootstrap.cc:492] T 2045e08a200148c69326351f38fd484d P 1b50a031f2784ba498b3e52e16916205: No bootstrap required, opened a new log
I20240502 01:54:10.909765 18612 tablet_bootstrap.cc:492] T 768e7cbf63a94d3a815ac77a4d0acf41 P da3584f429104d23a1a8899b35cc2138: Bootstrap starting.
I20240502 01:54:10.909870 18607 ts_tablet_manager.cc:1367] T 2045e08a200148c69326351f38fd484d P 1b50a031f2784ba498b3e52e16916205: Time spent bootstrapping tablet: real 0.006s	user 0.004s	sys 0.000s
I20240502 01:54:10.910730 18607 raft_consensus.cc:348] T 2045e08a200148c69326351f38fd484d P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } }
I20240502 01:54:10.910979 18607 raft_consensus.cc:374] T 2045e08a200148c69326351f38fd484d P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:10.911085 18607 raft_consensus.cc:729] T 2045e08a200148c69326351f38fd484d P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b50a031f2784ba498b3e52e16916205, State: Initialized, Role: FOLLOWER
I20240502 01:54:10.911370 18607 consensus_queue.cc:260] T 2045e08a200148c69326351f38fd484d P 1b50a031f2784ba498b3e52e16916205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } }
I20240502 01:54:10.912034 18607 ts_tablet_manager.cc:1398] T 2045e08a200148c69326351f38fd484d P 1b50a031f2784ba498b3e52e16916205: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240502 01:54:10.912537 18612 tablet_bootstrap.cc:654] T 768e7cbf63a94d3a815ac77a4d0acf41 P da3584f429104d23a1a8899b35cc2138: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:10.914479 18612 tablet_bootstrap.cc:492] T 768e7cbf63a94d3a815ac77a4d0acf41 P da3584f429104d23a1a8899b35cc2138: No bootstrap required, opened a new log
I20240502 01:54:10.914660 18612 ts_tablet_manager.cc:1367] T 768e7cbf63a94d3a815ac77a4d0acf41 P da3584f429104d23a1a8899b35cc2138: Time spent bootstrapping tablet: real 0.005s	user 0.003s	sys 0.000s
I20240502 01:54:10.915402 18612 raft_consensus.cc:348] T 768e7cbf63a94d3a815ac77a4d0acf41 P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } }
I20240502 01:54:10.915695 18612 raft_consensus.cc:374] T 768e7cbf63a94d3a815ac77a4d0acf41 P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:10.915796 18612 raft_consensus.cc:729] T 768e7cbf63a94d3a815ac77a4d0acf41 P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da3584f429104d23a1a8899b35cc2138, State: Initialized, Role: FOLLOWER
I20240502 01:54:10.916100 18612 consensus_queue.cc:260] T 768e7cbf63a94d3a815ac77a4d0acf41 P da3584f429104d23a1a8899b35cc2138 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } }
I20240502 01:54:10.916636 18612 ts_tablet_manager.cc:1398] T 768e7cbf63a94d3a815ac77a4d0acf41 P da3584f429104d23a1a8899b35cc2138: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 01:54:10.923635 18601 raft_consensus.cc:482] T 37eda67598fe491bbb29aa62b2c4bec7 P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:10.923863 18601 raft_consensus.cc:504] T 37eda67598fe491bbb29aa62b2c4bec7 P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } }
I20240502 01:54:10.924592 18601 leader_election.cc:288] T 37eda67598fe491bbb29aa62b2c4bec7 P 311f765f59754bdcb4eadc04e14bd4ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b50a031f2784ba498b3e52e16916205 (127.9.176.194:45617), 84d3a4b174c34e02aedab1e620731359 (127.9.176.195:37357)
I20240502 01:54:10.925236 17767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37eda67598fe491bbb29aa62b2c4bec7" candidate_uuid: "311f765f59754bdcb4eadc04e14bd4ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84d3a4b174c34e02aedab1e620731359" is_pre_election: true
I20240502 01:54:10.925238 17634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37eda67598fe491bbb29aa62b2c4bec7" candidate_uuid: "311f765f59754bdcb4eadc04e14bd4ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b50a031f2784ba498b3e52e16916205" is_pre_election: true
I20240502 01:54:10.925468 17634 raft_consensus.cc:2454] T 37eda67598fe491bbb29aa62b2c4bec7 P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 311f765f59754bdcb4eadc04e14bd4ea in term 0.
I20240502 01:54:10.925482 17767 raft_consensus.cc:2454] T 37eda67598fe491bbb29aa62b2c4bec7 P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 311f765f59754bdcb4eadc04e14bd4ea in term 0.
I20240502 01:54:10.925971 17435 leader_election.cc:302] T 37eda67598fe491bbb29aa62b2c4bec7 P 311f765f59754bdcb4eadc04e14bd4ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 311f765f59754bdcb4eadc04e14bd4ea, 84d3a4b174c34e02aedab1e620731359; no voters: 
I20240502 01:54:10.926267 18601 raft_consensus.cc:2781] T 37eda67598fe491bbb29aa62b2c4bec7 P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:10.926385 18601 raft_consensus.cc:482] T 37eda67598fe491bbb29aa62b2c4bec7 P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:10.926476 18601 raft_consensus.cc:3037] T 37eda67598fe491bbb29aa62b2c4bec7 P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:10.928570 18601 raft_consensus.cc:504] T 37eda67598fe491bbb29aa62b2c4bec7 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } }
I20240502 01:54:10.929230 18601 leader_election.cc:288] T 37eda67598fe491bbb29aa62b2c4bec7 P 311f765f59754bdcb4eadc04e14bd4ea [CANDIDATE]: Term 1 election: Requested vote from peers 1b50a031f2784ba498b3e52e16916205 (127.9.176.194:45617), 84d3a4b174c34e02aedab1e620731359 (127.9.176.195:37357)
I20240502 01:54:10.929944 17767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37eda67598fe491bbb29aa62b2c4bec7" candidate_uuid: "311f765f59754bdcb4eadc04e14bd4ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84d3a4b174c34e02aedab1e620731359"
I20240502 01:54:10.929996 17634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37eda67598fe491bbb29aa62b2c4bec7" candidate_uuid: "311f765f59754bdcb4eadc04e14bd4ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b50a031f2784ba498b3e52e16916205"
I20240502 01:54:10.930182 17767 raft_consensus.cc:3037] T 37eda67598fe491bbb29aa62b2c4bec7 P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:10.930219 17634 raft_consensus.cc:3037] T 37eda67598fe491bbb29aa62b2c4bec7 P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:10.932596 17767 raft_consensus.cc:2454] T 37eda67598fe491bbb29aa62b2c4bec7 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 311f765f59754bdcb4eadc04e14bd4ea in term 1.
I20240502 01:54:10.932619 17634 raft_consensus.cc:2454] T 37eda67598fe491bbb29aa62b2c4bec7 P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 311f765f59754bdcb4eadc04e14bd4ea in term 1.
I20240502 01:54:10.933099 17435 leader_election.cc:302] T 37eda67598fe491bbb29aa62b2c4bec7 P 311f765f59754bdcb4eadc04e14bd4ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 311f765f59754bdcb4eadc04e14bd4ea, 84d3a4b174c34e02aedab1e620731359; no voters: 
I20240502 01:54:10.933450 18601 raft_consensus.cc:2781] T 37eda67598fe491bbb29aa62b2c4bec7 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:10.933598 18601 raft_consensus.cc:686] T 37eda67598fe491bbb29aa62b2c4bec7 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 LEADER]: Becoming Leader. State: Replica: 311f765f59754bdcb4eadc04e14bd4ea, State: Running, Role: LEADER
I20240502 01:54:10.933909 18601 consensus_queue.cc:237] T 37eda67598fe491bbb29aa62b2c4bec7 P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } }
I20240502 01:54:10.936671 17353 catalog_manager.cc:5484] T 37eda67598fe491bbb29aa62b2c4bec7 P 311f765f59754bdcb4eadc04e14bd4ea reported cstate change: term changed from 0 to 1, leader changed from <none> to 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193). New cstate: current_term: 1 leader_uuid: "311f765f59754bdcb4eadc04e14bd4ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:11.016280 18536 raft_consensus.cc:482] T 768e7cbf63a94d3a815ac77a4d0acf41 P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:11.016498 18536 raft_consensus.cc:504] T 768e7cbf63a94d3a815ac77a4d0acf41 P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } }
I20240502 01:54:11.017361 18536 leader_election.cc:288] T 768e7cbf63a94d3a815ac77a4d0acf41 P 84d3a4b174c34e02aedab1e620731359 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b50a031f2784ba498b3e52e16916205 (127.9.176.194:45617), da3584f429104d23a1a8899b35cc2138 (127.9.176.196:46381)
I20240502 01:54:11.018069 17900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "768e7cbf63a94d3a815ac77a4d0acf41" candidate_uuid: "84d3a4b174c34e02aedab1e620731359" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da3584f429104d23a1a8899b35cc2138" is_pre_election: true
I20240502 01:54:11.018378 17900 raft_consensus.cc:2454] T 768e7cbf63a94d3a815ac77a4d0acf41 P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84d3a4b174c34e02aedab1e620731359 in term 0.
I20240502 01:54:11.018862 17700 leader_election.cc:302] T 768e7cbf63a94d3a815ac77a4d0acf41 P 84d3a4b174c34e02aedab1e620731359 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84d3a4b174c34e02aedab1e620731359, da3584f429104d23a1a8899b35cc2138; no voters: 
I20240502 01:54:11.019172 18536 raft_consensus.cc:2781] T 768e7cbf63a94d3a815ac77a4d0acf41 P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:11.019320 18536 raft_consensus.cc:482] T 768e7cbf63a94d3a815ac77a4d0acf41 P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:11.019424 18536 raft_consensus.cc:3037] T 768e7cbf63a94d3a815ac77a4d0acf41 P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:11.021670 18536 raft_consensus.cc:504] T 768e7cbf63a94d3a815ac77a4d0acf41 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } }
I20240502 01:54:11.022483 18536 leader_election.cc:288] T 768e7cbf63a94d3a815ac77a4d0acf41 P 84d3a4b174c34e02aedab1e620731359 [CANDIDATE]: Term 1 election: Requested vote from peers 1b50a031f2784ba498b3e52e16916205 (127.9.176.194:45617), da3584f429104d23a1a8899b35cc2138 (127.9.176.196:46381)
I20240502 01:54:11.023113 17900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "768e7cbf63a94d3a815ac77a4d0acf41" candidate_uuid: "84d3a4b174c34e02aedab1e620731359" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da3584f429104d23a1a8899b35cc2138"
I20240502 01:54:11.023371 17900 raft_consensus.cc:3037] T 768e7cbf63a94d3a815ac77a4d0acf41 P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:11.025017 17634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "768e7cbf63a94d3a815ac77a4d0acf41" candidate_uuid: "84d3a4b174c34e02aedab1e620731359" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b50a031f2784ba498b3e52e16916205" is_pre_election: true
I20240502 01:54:11.025175 17633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "768e7cbf63a94d3a815ac77a4d0acf41" candidate_uuid: "84d3a4b174c34e02aedab1e620731359" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b50a031f2784ba498b3e52e16916205"
I20240502 01:54:11.025280 17634 raft_consensus.cc:2454] T 768e7cbf63a94d3a815ac77a4d0acf41 P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84d3a4b174c34e02aedab1e620731359 in term 0.
I20240502 01:54:11.025974 17900 raft_consensus.cc:2454] T 768e7cbf63a94d3a815ac77a4d0acf41 P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84d3a4b174c34e02aedab1e620731359 in term 1.
I20240502 01:54:11.026363 17700 leader_election.cc:302] T 768e7cbf63a94d3a815ac77a4d0acf41 P 84d3a4b174c34e02aedab1e620731359 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 84d3a4b174c34e02aedab1e620731359, da3584f429104d23a1a8899b35cc2138; no voters: 1b50a031f2784ba498b3e52e16916205
I20240502 01:54:11.026670 18536 raft_consensus.cc:2781] T 768e7cbf63a94d3a815ac77a4d0acf41 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:11.027067 18536 raft_consensus.cc:686] T 768e7cbf63a94d3a815ac77a4d0acf41 P 84d3a4b174c34e02aedab1e620731359 [term 1 LEADER]: Becoming Leader. State: Replica: 84d3a4b174c34e02aedab1e620731359, State: Running, Role: LEADER
I20240502 01:54:11.027374 18536 consensus_queue.cc:237] T 768e7cbf63a94d3a815ac77a4d0acf41 P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } }
I20240502 01:54:11.030242 18536 raft_consensus.cc:482] T 2045e08a200148c69326351f38fd484d P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:11.030426 18536 raft_consensus.cc:504] T 2045e08a200148c69326351f38fd484d P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } }
I20240502 01:54:11.030331 17353 catalog_manager.cc:5484] T 768e7cbf63a94d3a815ac77a4d0acf41 P 84d3a4b174c34e02aedab1e620731359 reported cstate change: term changed from 0 to 1, leader changed from <none> to 84d3a4b174c34e02aedab1e620731359 (127.9.176.195). New cstate: current_term: 1 leader_uuid: "84d3a4b174c34e02aedab1e620731359" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:11.031296 18536 leader_election.cc:288] T 2045e08a200148c69326351f38fd484d P 84d3a4b174c34e02aedab1e620731359 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b50a031f2784ba498b3e52e16916205 (127.9.176.194:45617), da3584f429104d23a1a8899b35cc2138 (127.9.176.196:46381)
I20240502 01:54:11.031534 17633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2045e08a200148c69326351f38fd484d" candidate_uuid: "84d3a4b174c34e02aedab1e620731359" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b50a031f2784ba498b3e52e16916205" is_pre_election: true
I20240502 01:54:11.031787 17633 raft_consensus.cc:2454] T 2045e08a200148c69326351f38fd484d P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84d3a4b174c34e02aedab1e620731359 in term 0.
I20240502 01:54:11.031723 17900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2045e08a200148c69326351f38fd484d" candidate_uuid: "84d3a4b174c34e02aedab1e620731359" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da3584f429104d23a1a8899b35cc2138" is_pre_election: true
I20240502 01:54:11.031952 17900 raft_consensus.cc:2454] T 2045e08a200148c69326351f38fd484d P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84d3a4b174c34e02aedab1e620731359 in term 0.
I20240502 01:54:11.032367 17700 leader_election.cc:302] T 2045e08a200148c69326351f38fd484d P 84d3a4b174c34e02aedab1e620731359 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1b50a031f2784ba498b3e52e16916205, 84d3a4b174c34e02aedab1e620731359, da3584f429104d23a1a8899b35cc2138; no voters: 
I20240502 01:54:11.032733 18536 raft_consensus.cc:2781] T 2045e08a200148c69326351f38fd484d P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:11.032879 18536 raft_consensus.cc:482] T 2045e08a200148c69326351f38fd484d P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:11.033058 18536 raft_consensus.cc:3037] T 2045e08a200148c69326351f38fd484d P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:11.034977 18536 raft_consensus.cc:504] T 2045e08a200148c69326351f38fd484d P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } }
I20240502 01:54:11.035496 18536 leader_election.cc:288] T 2045e08a200148c69326351f38fd484d P 84d3a4b174c34e02aedab1e620731359 [CANDIDATE]: Term 1 election: Requested vote from peers 1b50a031f2784ba498b3e52e16916205 (127.9.176.194:45617), da3584f429104d23a1a8899b35cc2138 (127.9.176.196:46381)
I20240502 01:54:11.035957 17633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2045e08a200148c69326351f38fd484d" candidate_uuid: "84d3a4b174c34e02aedab1e620731359" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b50a031f2784ba498b3e52e16916205"
I20240502 01:54:11.036008 17900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2045e08a200148c69326351f38fd484d" candidate_uuid: "84d3a4b174c34e02aedab1e620731359" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da3584f429104d23a1a8899b35cc2138"
I20240502 01:54:11.036159 17633 raft_consensus.cc:3037] T 2045e08a200148c69326351f38fd484d P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:11.036206 17900 raft_consensus.cc:3037] T 2045e08a200148c69326351f38fd484d P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:11.038106 17900 raft_consensus.cc:2454] T 2045e08a200148c69326351f38fd484d P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84d3a4b174c34e02aedab1e620731359 in term 1.
I20240502 01:54:11.038089 17633 raft_consensus.cc:2454] T 2045e08a200148c69326351f38fd484d P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84d3a4b174c34e02aedab1e620731359 in term 1.
I20240502 01:54:11.038575 17700 leader_election.cc:302] T 2045e08a200148c69326351f38fd484d P 84d3a4b174c34e02aedab1e620731359 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84d3a4b174c34e02aedab1e620731359, da3584f429104d23a1a8899b35cc2138; no voters: 
I20240502 01:54:11.038941 18536 raft_consensus.cc:2781] T 2045e08a200148c69326351f38fd484d P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:11.039120 18536 raft_consensus.cc:686] T 2045e08a200148c69326351f38fd484d P 84d3a4b174c34e02aedab1e620731359 [term 1 LEADER]: Becoming Leader. State: Replica: 84d3a4b174c34e02aedab1e620731359, State: Running, Role: LEADER
I20240502 01:54:11.039386 18536 consensus_queue.cc:237] T 2045e08a200148c69326351f38fd484d P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } }
I20240502 01:54:11.042220 17353 catalog_manager.cc:5484] T 2045e08a200148c69326351f38fd484d P 84d3a4b174c34e02aedab1e620731359 reported cstate change: term changed from 0 to 1, leader changed from <none> to 84d3a4b174c34e02aedab1e620731359 (127.9.176.195). New cstate: current_term: 1 leader_uuid: "84d3a4b174c34e02aedab1e620731359" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:11.314306 18601 consensus_queue.cc:1035] T 37eda67598fe491bbb29aa62b2c4bec7 P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:11.321250 18601 consensus_queue.cc:1035] T 37eda67598fe491bbb29aa62b2c4bec7 P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:54:11.437575 18628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:54:11.437914 18628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:54:11.445812 18628 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240502 01:54:11.493516 17633 raft_consensus.cc:3037] T 768e7cbf63a94d3a815ac77a4d0acf41 P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:11.496973 18536 consensus_queue.cc:1035] T 768e7cbf63a94d3a815ac77a4d0acf41 P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:11.502547 18625 consensus_queue.cc:1035] T 768e7cbf63a94d3a815ac77a4d0acf41 P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:11.506242 18625 consensus_queue.cc:1035] T 2045e08a200148c69326351f38fd484d P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:11.517205 18536 consensus_queue.cc:1035] T 2045e08a200148c69326351f38fd484d P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 288.783 ms
  time per row: 0.144391 ms
I20240502 01:54:11.889631 17352 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56698:
name: "rebalance_test_table_2"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 01:54:11.899555 17478 tablet_service.cc:1467] Processing CreateTablet for tablet 3b3fa060e8d64911bec810ec07a358d2 (DEFAULT_TABLE table=rebalance_test_table_2 [id=bf67e64a43cd4367a7420c6bf1793faa]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:11.900159 17478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b3fa060e8d64911bec810ec07a358d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:11.900348 17879 tablet_service.cc:1467] Processing CreateTablet for tablet 3b3fa060e8d64911bec810ec07a358d2 (DEFAULT_TABLE table=rebalance_test_table_2 [id=bf67e64a43cd4367a7420c6bf1793faa]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:11.900784 17879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b3fa060e8d64911bec810ec07a358d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:11.901176 17745 tablet_service.cc:1467] Processing CreateTablet for tablet c91f74f46b04425aa5393515cb9f781c (DEFAULT_TABLE table=rebalance_test_table_2 [id=bf67e64a43cd4367a7420c6bf1793faa]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:11.901340 17611 tablet_service.cc:1467] Processing CreateTablet for tablet 91e0e82526c0489a8908e250bf3c851f (DEFAULT_TABLE table=rebalance_test_table_2 [id=bf67e64a43cd4367a7420c6bf1793faa]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:11.901639 17745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c91f74f46b04425aa5393515cb9f781c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:11.901760 17611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91e0e82526c0489a8908e250bf3c851f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:11.908164 17880 tablet_service.cc:1467] Processing CreateTablet for tablet c91f74f46b04425aa5393515cb9f781c (DEFAULT_TABLE table=rebalance_test_table_2 [id=bf67e64a43cd4367a7420c6bf1793faa]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:11.908679 17880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c91f74f46b04425aa5393515cb9f781c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:11.909031 17878 tablet_service.cc:1467] Processing CreateTablet for tablet 91e0e82526c0489a8908e250bf3c851f (DEFAULT_TABLE table=rebalance_test_table_2 [id=bf67e64a43cd4367a7420c6bf1793faa]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:11.909727 17878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91e0e82526c0489a8908e250bf3c851f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:11.912407 17612 tablet_service.cc:1467] Processing CreateTablet for tablet 3b3fa060e8d64911bec810ec07a358d2 (DEFAULT_TABLE table=rebalance_test_table_2 [id=bf67e64a43cd4367a7420c6bf1793faa]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:11.913096 17612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b3fa060e8d64911bec810ec07a358d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:11.914645 18701 tablet_bootstrap.cc:492] T 3b3fa060e8d64911bec810ec07a358d2 P 311f765f59754bdcb4eadc04e14bd4ea: Bootstrap starting.
I20240502 01:54:11.916735 18701 tablet_bootstrap.cc:654] T 3b3fa060e8d64911bec810ec07a358d2 P 311f765f59754bdcb4eadc04e14bd4ea: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:11.916785 17477 tablet_service.cc:1467] Processing CreateTablet for tablet c91f74f46b04425aa5393515cb9f781c (DEFAULT_TABLE table=rebalance_test_table_2 [id=bf67e64a43cd4367a7420c6bf1793faa]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:11.916815 17479 tablet_service.cc:1467] Processing CreateTablet for tablet 91e0e82526c0489a8908e250bf3c851f (DEFAULT_TABLE table=rebalance_test_table_2 [id=bf67e64a43cd4367a7420c6bf1793faa]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:11.917395 17479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91e0e82526c0489a8908e250bf3c851f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:11.919700 18701 tablet_bootstrap.cc:492] T 3b3fa060e8d64911bec810ec07a358d2 P 311f765f59754bdcb4eadc04e14bd4ea: No bootstrap required, opened a new log
I20240502 01:54:11.919977 18701 ts_tablet_manager.cc:1367] T 3b3fa060e8d64911bec810ec07a358d2 P 311f765f59754bdcb4eadc04e14bd4ea: Time spent bootstrapping tablet: real 0.005s	user 0.004s	sys 0.000s
I20240502 01:54:11.920604 18702 tablet_bootstrap.cc:492] T c91f74f46b04425aa5393515cb9f781c P 84d3a4b174c34e02aedab1e620731359: Bootstrap starting.
I20240502 01:54:11.920019 17477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c91f74f46b04425aa5393515cb9f781c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:11.921185 18701 raft_consensus.cc:348] T 3b3fa060e8d64911bec810ec07a358d2 P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } }
I20240502 01:54:11.921525 18701 raft_consensus.cc:374] T 3b3fa060e8d64911bec810ec07a358d2 P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:11.921682 18701 raft_consensus.cc:729] T 3b3fa060e8d64911bec810ec07a358d2 P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 311f765f59754bdcb4eadc04e14bd4ea, State: Initialized, Role: FOLLOWER
I20240502 01:54:11.921768 18703 tablet_bootstrap.cc:492] T 91e0e82526c0489a8908e250bf3c851f P da3584f429104d23a1a8899b35cc2138: Bootstrap starting.
I20240502 01:54:11.922046 18701 consensus_queue.cc:260] T 3b3fa060e8d64911bec810ec07a358d2 P 311f765f59754bdcb4eadc04e14bd4ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } }
I20240502 01:54:11.922837 18701 ts_tablet_manager.cc:1398] T 3b3fa060e8d64911bec810ec07a358d2 P 311f765f59754bdcb4eadc04e14bd4ea: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 01:54:11.923352 18702 tablet_bootstrap.cc:654] T c91f74f46b04425aa5393515cb9f781c P 84d3a4b174c34e02aedab1e620731359: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:11.924541 18703 tablet_bootstrap.cc:654] T 91e0e82526c0489a8908e250bf3c851f P da3584f429104d23a1a8899b35cc2138: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:11.929229 18703 tablet_bootstrap.cc:492] T 91e0e82526c0489a8908e250bf3c851f P da3584f429104d23a1a8899b35cc2138: No bootstrap required, opened a new log
I20240502 01:54:11.929561 18703 ts_tablet_manager.cc:1367] T 91e0e82526c0489a8908e250bf3c851f P da3584f429104d23a1a8899b35cc2138: Time spent bootstrapping tablet: real 0.008s	user 0.000s	sys 0.007s
I20240502 01:54:11.930914 18703 raft_consensus.cc:348] T 91e0e82526c0489a8908e250bf3c851f P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } }
I20240502 01:54:11.932113 18703 raft_consensus.cc:374] T 91e0e82526c0489a8908e250bf3c851f P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:11.932309 18703 raft_consensus.cc:729] T 91e0e82526c0489a8908e250bf3c851f P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da3584f429104d23a1a8899b35cc2138, State: Initialized, Role: FOLLOWER
I20240502 01:54:11.932770 18703 consensus_queue.cc:260] T 91e0e82526c0489a8908e250bf3c851f P da3584f429104d23a1a8899b35cc2138 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } }
I20240502 01:54:11.936486 18701 tablet_bootstrap.cc:492] T c91f74f46b04425aa5393515cb9f781c P 311f765f59754bdcb4eadc04e14bd4ea: Bootstrap starting.
I20240502 01:54:11.937606 18703 ts_tablet_manager.cc:1398] T 91e0e82526c0489a8908e250bf3c851f P da3584f429104d23a1a8899b35cc2138: Time spent starting tablet: real 0.008s	user 0.000s	sys 0.003s
I20240502 01:54:11.937954 18703 tablet_bootstrap.cc:492] T 3b3fa060e8d64911bec810ec07a358d2 P da3584f429104d23a1a8899b35cc2138: Bootstrap starting.
I20240502 01:54:11.938251 18701 tablet_bootstrap.cc:654] T c91f74f46b04425aa5393515cb9f781c P 311f765f59754bdcb4eadc04e14bd4ea: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:11.941265 18707 tablet_bootstrap.cc:492] T 3b3fa060e8d64911bec810ec07a358d2 P 1b50a031f2784ba498b3e52e16916205: Bootstrap starting.
I20240502 01:54:11.942056 18702 tablet_bootstrap.cc:492] T c91f74f46b04425aa5393515cb9f781c P 84d3a4b174c34e02aedab1e620731359: No bootstrap required, opened a new log
I20240502 01:54:11.942246 18702 ts_tablet_manager.cc:1367] T c91f74f46b04425aa5393515cb9f781c P 84d3a4b174c34e02aedab1e620731359: Time spent bootstrapping tablet: real 0.022s	user 0.004s	sys 0.012s
I20240502 01:54:11.943377 18702 raft_consensus.cc:348] T c91f74f46b04425aa5393515cb9f781c P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } }
I20240502 01:54:11.943655 18702 raft_consensus.cc:374] T c91f74f46b04425aa5393515cb9f781c P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:11.943763 18702 raft_consensus.cc:729] T c91f74f46b04425aa5393515cb9f781c P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84d3a4b174c34e02aedab1e620731359, State: Initialized, Role: FOLLOWER
I20240502 01:54:11.944070 18702 consensus_queue.cc:260] T c91f74f46b04425aa5393515cb9f781c P 84d3a4b174c34e02aedab1e620731359 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } }
I20240502 01:54:11.944723 18702 ts_tablet_manager.cc:1398] T c91f74f46b04425aa5393515cb9f781c P 84d3a4b174c34e02aedab1e620731359: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 01:54:11.946245 18707 tablet_bootstrap.cc:654] T 3b3fa060e8d64911bec810ec07a358d2 P 1b50a031f2784ba498b3e52e16916205: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:11.946509 18703 tablet_bootstrap.cc:654] T 3b3fa060e8d64911bec810ec07a358d2 P da3584f429104d23a1a8899b35cc2138: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:11.947297 18701 tablet_bootstrap.cc:492] T c91f74f46b04425aa5393515cb9f781c P 311f765f59754bdcb4eadc04e14bd4ea: No bootstrap required, opened a new log
I20240502 01:54:11.947537 18701 ts_tablet_manager.cc:1367] T c91f74f46b04425aa5393515cb9f781c P 311f765f59754bdcb4eadc04e14bd4ea: Time spent bootstrapping tablet: real 0.011s	user 0.001s	sys 0.009s
I20240502 01:54:11.948568 18701 raft_consensus.cc:348] T c91f74f46b04425aa5393515cb9f781c P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } }
I20240502 01:54:11.948880 18707 tablet_bootstrap.cc:492] T 3b3fa060e8d64911bec810ec07a358d2 P 1b50a031f2784ba498b3e52e16916205: No bootstrap required, opened a new log
I20240502 01:54:11.948874 18701 raft_consensus.cc:374] T c91f74f46b04425aa5393515cb9f781c P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:11.948963 18701 raft_consensus.cc:729] T c91f74f46b04425aa5393515cb9f781c P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 311f765f59754bdcb4eadc04e14bd4ea, State: Initialized, Role: FOLLOWER
I20240502 01:54:11.949090 18707 ts_tablet_manager.cc:1367] T 3b3fa060e8d64911bec810ec07a358d2 P 1b50a031f2784ba498b3e52e16916205: Time spent bootstrapping tablet: real 0.008s	user 0.003s	sys 0.002s
I20240502 01:54:11.949299 18701 consensus_queue.cc:260] T c91f74f46b04425aa5393515cb9f781c P 311f765f59754bdcb4eadc04e14bd4ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } }
I20240502 01:54:11.950110 18701 ts_tablet_manager.cc:1398] T c91f74f46b04425aa5393515cb9f781c P 311f765f59754bdcb4eadc04e14bd4ea: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240502 01:54:11.950083 18707 raft_consensus.cc:348] T 3b3fa060e8d64911bec810ec07a358d2 P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } }
I20240502 01:54:11.950336 18707 raft_consensus.cc:374] T 3b3fa060e8d64911bec810ec07a358d2 P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:11.950748 18701 tablet_bootstrap.cc:492] T 91e0e82526c0489a8908e250bf3c851f P 311f765f59754bdcb4eadc04e14bd4ea: Bootstrap starting.
I20240502 01:54:11.950459 18707 raft_consensus.cc:729] T 3b3fa060e8d64911bec810ec07a358d2 P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b50a031f2784ba498b3e52e16916205, State: Initialized, Role: FOLLOWER
I20240502 01:54:11.951630 18707 consensus_queue.cc:260] T 3b3fa060e8d64911bec810ec07a358d2 P 1b50a031f2784ba498b3e52e16916205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } }
I20240502 01:54:11.952616 18701 tablet_bootstrap.cc:654] T 91e0e82526c0489a8908e250bf3c851f P 311f765f59754bdcb4eadc04e14bd4ea: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:11.953521 18707 ts_tablet_manager.cc:1398] T 3b3fa060e8d64911bec810ec07a358d2 P 1b50a031f2784ba498b3e52e16916205: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.000s
I20240502 01:54:11.953914 18707 tablet_bootstrap.cc:492] T 91e0e82526c0489a8908e250bf3c851f P 1b50a031f2784ba498b3e52e16916205: Bootstrap starting.
I20240502 01:54:11.955627 18703 tablet_bootstrap.cc:492] T 3b3fa060e8d64911bec810ec07a358d2 P da3584f429104d23a1a8899b35cc2138: No bootstrap required, opened a new log
I20240502 01:54:11.955842 18703 ts_tablet_manager.cc:1367] T 3b3fa060e8d64911bec810ec07a358d2 P da3584f429104d23a1a8899b35cc2138: Time spent bootstrapping tablet: real 0.018s	user 0.001s	sys 0.007s
I20240502 01:54:11.956882 18703 raft_consensus.cc:348] T 3b3fa060e8d64911bec810ec07a358d2 P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } }
I20240502 01:54:11.957139 18703 raft_consensus.cc:374] T 3b3fa060e8d64911bec810ec07a358d2 P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:11.957151 18615 raft_consensus.cc:482] T 3b3fa060e8d64911bec810ec07a358d2 P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:11.957221 18703 raft_consensus.cc:729] T 3b3fa060e8d64911bec810ec07a358d2 P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da3584f429104d23a1a8899b35cc2138, State: Initialized, Role: FOLLOWER
I20240502 01:54:11.957320 18615 raft_consensus.cc:504] T 3b3fa060e8d64911bec810ec07a358d2 P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } }
I20240502 01:54:11.957495 18703 consensus_queue.cc:260] T 3b3fa060e8d64911bec810ec07a358d2 P da3584f429104d23a1a8899b35cc2138 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } }
I20240502 01:54:11.958042 18703 ts_tablet_manager.cc:1398] T 3b3fa060e8d64911bec810ec07a358d2 P da3584f429104d23a1a8899b35cc2138: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 01:54:11.958407 18703 tablet_bootstrap.cc:492] T c91f74f46b04425aa5393515cb9f781c P da3584f429104d23a1a8899b35cc2138: Bootstrap starting.
I20240502 01:54:11.959466 18615 leader_election.cc:288] T 3b3fa060e8d64911bec810ec07a358d2 P 1b50a031f2784ba498b3e52e16916205 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193:35397), da3584f429104d23a1a8899b35cc2138 (127.9.176.196:46381)
I20240502 01:54:11.960044 18701 tablet_bootstrap.cc:492] T 91e0e82526c0489a8908e250bf3c851f P 311f765f59754bdcb4eadc04e14bd4ea: No bootstrap required, opened a new log
I20240502 01:54:11.960167 18707 tablet_bootstrap.cc:654] T 91e0e82526c0489a8908e250bf3c851f P 1b50a031f2784ba498b3e52e16916205: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:11.960215 18703 tablet_bootstrap.cc:654] T c91f74f46b04425aa5393515cb9f781c P da3584f429104d23a1a8899b35cc2138: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:11.960228 18701 ts_tablet_manager.cc:1367] T 91e0e82526c0489a8908e250bf3c851f P 311f765f59754bdcb4eadc04e14bd4ea: Time spent bootstrapping tablet: real 0.010s	user 0.000s	sys 0.005s
I20240502 01:54:11.961135 18701 raft_consensus.cc:348] T 91e0e82526c0489a8908e250bf3c851f P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } }
I20240502 01:54:11.961458 18701 raft_consensus.cc:374] T 91e0e82526c0489a8908e250bf3c851f P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:11.961567 18701 raft_consensus.cc:729] T 91e0e82526c0489a8908e250bf3c851f P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 311f765f59754bdcb4eadc04e14bd4ea, State: Initialized, Role: FOLLOWER
I20240502 01:54:11.961894 18701 consensus_queue.cc:260] T 91e0e82526c0489a8908e250bf3c851f P 311f765f59754bdcb4eadc04e14bd4ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } }
I20240502 01:54:11.963423 18701 ts_tablet_manager.cc:1398] T 91e0e82526c0489a8908e250bf3c851f P 311f765f59754bdcb4eadc04e14bd4ea: Time spent starting tablet: real 0.003s	user 0.003s	sys 0.000s
I20240502 01:54:11.978493 18703 tablet_bootstrap.cc:492] T c91f74f46b04425aa5393515cb9f781c P da3584f429104d23a1a8899b35cc2138: No bootstrap required, opened a new log
I20240502 01:54:11.978717 18703 ts_tablet_manager.cc:1367] T c91f74f46b04425aa5393515cb9f781c P da3584f429104d23a1a8899b35cc2138: Time spent bootstrapping tablet: real 0.020s	user 0.018s	sys 0.000s
I20240502 01:54:11.979925 18707 tablet_bootstrap.cc:492] T 91e0e82526c0489a8908e250bf3c851f P 1b50a031f2784ba498b3e52e16916205: No bootstrap required, opened a new log
I20240502 01:54:11.979805 18703 raft_consensus.cc:348] T c91f74f46b04425aa5393515cb9f781c P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } }
I20240502 01:54:11.980046 18703 raft_consensus.cc:374] T c91f74f46b04425aa5393515cb9f781c P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:11.980129 18707 ts_tablet_manager.cc:1367] T 91e0e82526c0489a8908e250bf3c851f P 1b50a031f2784ba498b3e52e16916205: Time spent bootstrapping tablet: real 0.026s	user 0.011s	sys 0.005s
I20240502 01:54:11.980152 18703 raft_consensus.cc:729] T c91f74f46b04425aa5393515cb9f781c P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da3584f429104d23a1a8899b35cc2138, State: Initialized, Role: FOLLOWER
I20240502 01:54:11.980481 18703 consensus_queue.cc:260] T c91f74f46b04425aa5393515cb9f781c P da3584f429104d23a1a8899b35cc2138 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } }
I20240502 01:54:11.981902 18703 ts_tablet_manager.cc:1398] T c91f74f46b04425aa5393515cb9f781c P da3584f429104d23a1a8899b35cc2138: Time spent starting tablet: real 0.003s	user 0.003s	sys 0.000s
I20240502 01:54:11.983774 18707 raft_consensus.cc:348] T 91e0e82526c0489a8908e250bf3c851f P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } }
I20240502 01:54:11.984092 18707 raft_consensus.cc:374] T 91e0e82526c0489a8908e250bf3c851f P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:11.984208 18707 raft_consensus.cc:729] T 91e0e82526c0489a8908e250bf3c851f P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b50a031f2784ba498b3e52e16916205, State: Initialized, Role: FOLLOWER
I20240502 01:54:11.984567 18707 consensus_queue.cc:260] T 91e0e82526c0489a8908e250bf3c851f P 1b50a031f2784ba498b3e52e16916205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } }
I20240502 01:54:11.985877 18707 ts_tablet_manager.cc:1398] T 91e0e82526c0489a8908e250bf3c851f P 1b50a031f2784ba498b3e52e16916205: Time spent starting tablet: real 0.006s	user 0.003s	sys 0.000s
I20240502 01:54:11.986364 17900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b3fa060e8d64911bec810ec07a358d2" candidate_uuid: "1b50a031f2784ba498b3e52e16916205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da3584f429104d23a1a8899b35cc2138" is_pre_election: true
I20240502 01:54:11.986600 17500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b3fa060e8d64911bec810ec07a358d2" candidate_uuid: "1b50a031f2784ba498b3e52e16916205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "311f765f59754bdcb4eadc04e14bd4ea" is_pre_election: true
I20240502 01:54:11.986666 17900 raft_consensus.cc:2454] T 3b3fa060e8d64911bec810ec07a358d2 P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b50a031f2784ba498b3e52e16916205 in term 0.
I20240502 01:54:11.986902 17500 raft_consensus.cc:2454] T 3b3fa060e8d64911bec810ec07a358d2 P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b50a031f2784ba498b3e52e16916205 in term 0.
I20240502 01:54:11.987460 17565 leader_election.cc:302] T 3b3fa060e8d64911bec810ec07a358d2 P 1b50a031f2784ba498b3e52e16916205 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b50a031f2784ba498b3e52e16916205, da3584f429104d23a1a8899b35cc2138; no voters: 
I20240502 01:54:11.987951 18615 raft_consensus.cc:2781] T 3b3fa060e8d64911bec810ec07a358d2 P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:11.988097 18615 raft_consensus.cc:482] T 3b3fa060e8d64911bec810ec07a358d2 P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:11.988193 18615 raft_consensus.cc:3037] T 3b3fa060e8d64911bec810ec07a358d2 P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:11.990510 18615 raft_consensus.cc:504] T 3b3fa060e8d64911bec810ec07a358d2 P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } }
I20240502 01:54:11.991176 18615 leader_election.cc:288] T 3b3fa060e8d64911bec810ec07a358d2 P 1b50a031f2784ba498b3e52e16916205 [CANDIDATE]: Term 1 election: Requested vote from peers 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193:35397), da3584f429104d23a1a8899b35cc2138 (127.9.176.196:46381)
I20240502 01:54:11.991724 17500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b3fa060e8d64911bec810ec07a358d2" candidate_uuid: "1b50a031f2784ba498b3e52e16916205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "311f765f59754bdcb4eadc04e14bd4ea"
I20240502 01:54:11.991747 17900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b3fa060e8d64911bec810ec07a358d2" candidate_uuid: "1b50a031f2784ba498b3e52e16916205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da3584f429104d23a1a8899b35cc2138"
I20240502 01:54:11.991958 17500 raft_consensus.cc:3037] T 3b3fa060e8d64911bec810ec07a358d2 P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:11.991981 17900 raft_consensus.cc:3037] T 3b3fa060e8d64911bec810ec07a358d2 P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:11.994419 17500 raft_consensus.cc:2454] T 3b3fa060e8d64911bec810ec07a358d2 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b50a031f2784ba498b3e52e16916205 in term 1.
I20240502 01:54:11.994489 17900 raft_consensus.cc:2454] T 3b3fa060e8d64911bec810ec07a358d2 P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b50a031f2784ba498b3e52e16916205 in term 1.
I20240502 01:54:11.994971 17568 leader_election.cc:302] T 3b3fa060e8d64911bec810ec07a358d2 P 1b50a031f2784ba498b3e52e16916205 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b50a031f2784ba498b3e52e16916205, 311f765f59754bdcb4eadc04e14bd4ea; no voters: 
I20240502 01:54:11.995308 18615 raft_consensus.cc:2781] T 3b3fa060e8d64911bec810ec07a358d2 P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:11.995831 18615 raft_consensus.cc:686] T 3b3fa060e8d64911bec810ec07a358d2 P 1b50a031f2784ba498b3e52e16916205 [term 1 LEADER]: Becoming Leader. State: Replica: 1b50a031f2784ba498b3e52e16916205, State: Running, Role: LEADER
I20240502 01:54:11.996395 18615 consensus_queue.cc:237] T 3b3fa060e8d64911bec810ec07a358d2 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } }
I20240502 01:54:12.000085 17352 catalog_manager.cc:5484] T 3b3fa060e8d64911bec810ec07a358d2 P 1b50a031f2784ba498b3e52e16916205 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b50a031f2784ba498b3e52e16916205 (127.9.176.194). New cstate: current_term: 1 leader_uuid: "1b50a031f2784ba498b3e52e16916205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:12.036080 18677 raft_consensus.cc:482] T c91f74f46b04425aa5393515cb9f781c P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:12.036285 18677 raft_consensus.cc:504] T c91f74f46b04425aa5393515cb9f781c P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } }
I20240502 01:54:12.037070 18677 leader_election.cc:288] T c91f74f46b04425aa5393515cb9f781c P 84d3a4b174c34e02aedab1e620731359 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193:35397), da3584f429104d23a1a8899b35cc2138 (127.9.176.196:46381)
I20240502 01:54:12.037683 17500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c91f74f46b04425aa5393515cb9f781c" candidate_uuid: "84d3a4b174c34e02aedab1e620731359" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "311f765f59754bdcb4eadc04e14bd4ea" is_pre_election: true
I20240502 01:54:12.037710 17900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c91f74f46b04425aa5393515cb9f781c" candidate_uuid: "84d3a4b174c34e02aedab1e620731359" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da3584f429104d23a1a8899b35cc2138" is_pre_election: true
I20240502 01:54:12.037907 17500 raft_consensus.cc:2454] T c91f74f46b04425aa5393515cb9f781c P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84d3a4b174c34e02aedab1e620731359 in term 0.
I20240502 01:54:12.037951 17900 raft_consensus.cc:2454] T c91f74f46b04425aa5393515cb9f781c P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84d3a4b174c34e02aedab1e620731359 in term 0.
I20240502 01:54:12.038375 17703 leader_election.cc:302] T c91f74f46b04425aa5393515cb9f781c P 84d3a4b174c34e02aedab1e620731359 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 311f765f59754bdcb4eadc04e14bd4ea, 84d3a4b174c34e02aedab1e620731359; no voters: 
I20240502 01:54:12.038681 18677 raft_consensus.cc:2781] T c91f74f46b04425aa5393515cb9f781c P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:12.038800 18677 raft_consensus.cc:482] T c91f74f46b04425aa5393515cb9f781c P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:12.038904 18677 raft_consensus.cc:3037] T c91f74f46b04425aa5393515cb9f781c P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:12.041054 18677 raft_consensus.cc:504] T c91f74f46b04425aa5393515cb9f781c P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } }
I20240502 01:54:12.041651 18677 leader_election.cc:288] T c91f74f46b04425aa5393515cb9f781c P 84d3a4b174c34e02aedab1e620731359 [CANDIDATE]: Term 1 election: Requested vote from peers 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193:35397), da3584f429104d23a1a8899b35cc2138 (127.9.176.196:46381)
I20240502 01:54:12.042253 17900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c91f74f46b04425aa5393515cb9f781c" candidate_uuid: "84d3a4b174c34e02aedab1e620731359" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da3584f429104d23a1a8899b35cc2138"
I20240502 01:54:12.042291 17500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c91f74f46b04425aa5393515cb9f781c" candidate_uuid: "84d3a4b174c34e02aedab1e620731359" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "311f765f59754bdcb4eadc04e14bd4ea"
I20240502 01:54:12.042457 17900 raft_consensus.cc:3037] T c91f74f46b04425aa5393515cb9f781c P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:12.042483 17500 raft_consensus.cc:3037] T c91f74f46b04425aa5393515cb9f781c P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:12.044521 17900 raft_consensus.cc:2454] T c91f74f46b04425aa5393515cb9f781c P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84d3a4b174c34e02aedab1e620731359 in term 1.
I20240502 01:54:12.044652 17500 raft_consensus.cc:2454] T c91f74f46b04425aa5393515cb9f781c P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84d3a4b174c34e02aedab1e620731359 in term 1.
I20240502 01:54:12.045104 17700 leader_election.cc:302] T c91f74f46b04425aa5393515cb9f781c P 84d3a4b174c34e02aedab1e620731359 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84d3a4b174c34e02aedab1e620731359, da3584f429104d23a1a8899b35cc2138; no voters: 
I20240502 01:54:12.045491 18677 raft_consensus.cc:2781] T c91f74f46b04425aa5393515cb9f781c P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:12.045656 18677 raft_consensus.cc:686] T c91f74f46b04425aa5393515cb9f781c P 84d3a4b174c34e02aedab1e620731359 [term 1 LEADER]: Becoming Leader. State: Replica: 84d3a4b174c34e02aedab1e620731359, State: Running, Role: LEADER
I20240502 01:54:12.045959 18677 consensus_queue.cc:237] T c91f74f46b04425aa5393515cb9f781c P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } }
I20240502 01:54:12.049027 17352 catalog_manager.cc:5484] T c91f74f46b04425aa5393515cb9f781c P 84d3a4b174c34e02aedab1e620731359 reported cstate change: term changed from 0 to 1, leader changed from <none> to 84d3a4b174c34e02aedab1e620731359 (127.9.176.195). New cstate: current_term: 1 leader_uuid: "84d3a4b174c34e02aedab1e620731359" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:12.061060 18615 raft_consensus.cc:482] T 91e0e82526c0489a8908e250bf3c851f P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:12.061286 18615 raft_consensus.cc:504] T 91e0e82526c0489a8908e250bf3c851f P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } }
I20240502 01:54:12.061964 18615 leader_election.cc:288] T 91e0e82526c0489a8908e250bf3c851f P 1b50a031f2784ba498b3e52e16916205 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193:35397), da3584f429104d23a1a8899b35cc2138 (127.9.176.196:46381)
I20240502 01:54:12.062719 17500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e0e82526c0489a8908e250bf3c851f" candidate_uuid: "1b50a031f2784ba498b3e52e16916205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "311f765f59754bdcb4eadc04e14bd4ea" is_pre_election: true
I20240502 01:54:12.062736 17900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e0e82526c0489a8908e250bf3c851f" candidate_uuid: "1b50a031f2784ba498b3e52e16916205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da3584f429104d23a1a8899b35cc2138" is_pre_election: true
I20240502 01:54:12.063020 17900 raft_consensus.cc:2454] T 91e0e82526c0489a8908e250bf3c851f P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b50a031f2784ba498b3e52e16916205 in term 0.
I20240502 01:54:12.063042 17500 raft_consensus.cc:2454] T 91e0e82526c0489a8908e250bf3c851f P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b50a031f2784ba498b3e52e16916205 in term 0.
I20240502 01:54:12.063529 17568 leader_election.cc:302] T 91e0e82526c0489a8908e250bf3c851f P 1b50a031f2784ba498b3e52e16916205 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b50a031f2784ba498b3e52e16916205, 311f765f59754bdcb4eadc04e14bd4ea; no voters: 
I20240502 01:54:12.063952 18615 raft_consensus.cc:2781] T 91e0e82526c0489a8908e250bf3c851f P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:12.064070 18615 raft_consensus.cc:482] T 91e0e82526c0489a8908e250bf3c851f P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:12.064215 18615 raft_consensus.cc:3037] T 91e0e82526c0489a8908e250bf3c851f P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:12.066272 18615 raft_consensus.cc:504] T 91e0e82526c0489a8908e250bf3c851f P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } }
I20240502 01:54:12.066972 18615 leader_election.cc:288] T 91e0e82526c0489a8908e250bf3c851f P 1b50a031f2784ba498b3e52e16916205 [CANDIDATE]: Term 1 election: Requested vote from peers 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193:35397), da3584f429104d23a1a8899b35cc2138 (127.9.176.196:46381)
I20240502 01:54:12.067629 17900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e0e82526c0489a8908e250bf3c851f" candidate_uuid: "1b50a031f2784ba498b3e52e16916205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da3584f429104d23a1a8899b35cc2138"
I20240502 01:54:12.067629 17500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e0e82526c0489a8908e250bf3c851f" candidate_uuid: "1b50a031f2784ba498b3e52e16916205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "311f765f59754bdcb4eadc04e14bd4ea"
I20240502 01:54:12.067896 17900 raft_consensus.cc:3037] T 91e0e82526c0489a8908e250bf3c851f P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:12.067893 17500 raft_consensus.cc:3037] T 91e0e82526c0489a8908e250bf3c851f P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:12.070921 17900 raft_consensus.cc:2454] T 91e0e82526c0489a8908e250bf3c851f P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b50a031f2784ba498b3e52e16916205 in term 1.
I20240502 01:54:12.070921 17500 raft_consensus.cc:2454] T 91e0e82526c0489a8908e250bf3c851f P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b50a031f2784ba498b3e52e16916205 in term 1.
I20240502 01:54:12.071472 17568 leader_election.cc:302] T 91e0e82526c0489a8908e250bf3c851f P 1b50a031f2784ba498b3e52e16916205 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b50a031f2784ba498b3e52e16916205, 311f765f59754bdcb4eadc04e14bd4ea; no voters: 
I20240502 01:54:12.071873 18615 raft_consensus.cc:2781] T 91e0e82526c0489a8908e250bf3c851f P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:12.072046 18615 raft_consensus.cc:686] T 91e0e82526c0489a8908e250bf3c851f P 1b50a031f2784ba498b3e52e16916205 [term 1 LEADER]: Becoming Leader. State: Replica: 1b50a031f2784ba498b3e52e16916205, State: Running, Role: LEADER
I20240502 01:54:12.072350 18615 consensus_queue.cc:237] T 91e0e82526c0489a8908e250bf3c851f P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } }
I20240502 01:54:12.075244 17352 catalog_manager.cc:5484] T 91e0e82526c0489a8908e250bf3c851f P 1b50a031f2784ba498b3e52e16916205 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b50a031f2784ba498b3e52e16916205 (127.9.176.194). New cstate: current_term: 1 leader_uuid: "1b50a031f2784ba498b3e52e16916205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:12.425388 18680 consensus_queue.cc:1035] T c91f74f46b04425aa5393515cb9f781c P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:12.435333 18680 consensus_queue.cc:1035] T c91f74f46b04425aa5393515cb9f781c P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:54:12.489773 18721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:54:12.490195 18721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:54:12.498560 18721 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 01:54:12.524003 18615 consensus_queue.cc:1035] T 3b3fa060e8d64911bec810ec07a358d2 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:12.529556 18615 consensus_queue.cc:1035] T 3b3fa060e8d64911bec810ec07a358d2 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:12.533144 18615 consensus_queue.cc:1035] T 91e0e82526c0489a8908e250bf3c851f P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_2'
I20240502 01:54:12.538043 18717 consensus_queue.cc:1035] T 91e0e82526c0489a8908e250bf3c851f P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 265.225 ms
  time per row: 0.132612 ms
I20240502 01:54:12.924356 17355 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56698:
name: "rebalance_test_table_3"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 01:54:12.933149 17479 tablet_service.cc:1467] Processing CreateTablet for tablet 88a137bcfc76410583f910b377d1be7f (DEFAULT_TABLE table=rebalance_test_table_3 [id=c5e1575e337a4256a90ce5b685b3ff28]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:12.933418 17745 tablet_service.cc:1467] Processing CreateTablet for tablet 88a137bcfc76410583f910b377d1be7f (DEFAULT_TABLE table=rebalance_test_table_3 [id=c5e1575e337a4256a90ce5b685b3ff28]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:12.933667 17479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88a137bcfc76410583f910b377d1be7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:12.933851 17745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88a137bcfc76410583f910b377d1be7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:12.933915 17612 tablet_service.cc:1467] Processing CreateTablet for tablet 88a137bcfc76410583f910b377d1be7f (DEFAULT_TABLE table=rebalance_test_table_3 [id=c5e1575e337a4256a90ce5b685b3ff28]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:12.934372 17612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88a137bcfc76410583f910b377d1be7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:12.935297 17747 tablet_service.cc:1467] Processing CreateTablet for tablet 3ca5c4988ff74e3ca8d7a5614bcb93e2 (DEFAULT_TABLE table=rebalance_test_table_3 [id=c5e1575e337a4256a90ce5b685b3ff28]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:12.936508 17880 tablet_service.cc:1467] Processing CreateTablet for tablet 3ca5c4988ff74e3ca8d7a5614bcb93e2 (DEFAULT_TABLE table=rebalance_test_table_3 [id=c5e1575e337a4256a90ce5b685b3ff28]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:12.936856 17747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ca5c4988ff74e3ca8d7a5614bcb93e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:12.936975 17880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ca5c4988ff74e3ca8d7a5614bcb93e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:12.937361 17746 tablet_service.cc:1467] Processing CreateTablet for tablet 3bdd49939e994240a6d15dbd11559ff4 (DEFAULT_TABLE table=rebalance_test_table_3 [id=c5e1575e337a4256a90ce5b685b3ff28]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:12.937705 17746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bdd49939e994240a6d15dbd11559ff4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:12.937700 17611 tablet_service.cc:1467] Processing CreateTablet for tablet 3bdd49939e994240a6d15dbd11559ff4 (DEFAULT_TABLE table=rebalance_test_table_3 [id=c5e1575e337a4256a90ce5b685b3ff28]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:12.937813 17878 tablet_service.cc:1467] Processing CreateTablet for tablet 3bdd49939e994240a6d15dbd11559ff4 (DEFAULT_TABLE table=rebalance_test_table_3 [id=c5e1575e337a4256a90ce5b685b3ff28]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:12.938119 17611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bdd49939e994240a6d15dbd11559ff4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:12.938170 17878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bdd49939e994240a6d15dbd11559ff4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:12.944880 17477 tablet_service.cc:1467] Processing CreateTablet for tablet 3ca5c4988ff74e3ca8d7a5614bcb93e2 (DEFAULT_TABLE table=rebalance_test_table_3 [id=c5e1575e337a4256a90ce5b685b3ff28]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:12.945392 17477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ca5c4988ff74e3ca8d7a5614bcb93e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:12.946072 18785 tablet_bootstrap.cc:492] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P da3584f429104d23a1a8899b35cc2138: Bootstrap starting.
I20240502 01:54:12.948012 18785 tablet_bootstrap.cc:654] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P da3584f429104d23a1a8899b35cc2138: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:12.950553 18785 tablet_bootstrap.cc:492] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P da3584f429104d23a1a8899b35cc2138: No bootstrap required, opened a new log
I20240502 01:54:12.950721 18785 ts_tablet_manager.cc:1367] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P da3584f429104d23a1a8899b35cc2138: Time spent bootstrapping tablet: real 0.005s	user 0.004s	sys 0.000s
I20240502 01:54:12.951628 18785 raft_consensus.cc:348] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } }
I20240502 01:54:12.952003 18785 raft_consensus.cc:374] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:12.952178 18785 raft_consensus.cc:729] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da3584f429104d23a1a8899b35cc2138, State: Initialized, Role: FOLLOWER
I20240502 01:54:12.952524 18785 consensus_queue.cc:260] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P da3584f429104d23a1a8899b35cc2138 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } }
I20240502 01:54:12.953979 18785 ts_tablet_manager.cc:1398] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P da3584f429104d23a1a8899b35cc2138: Time spent starting tablet: real 0.003s	user 0.003s	sys 0.000s
I20240502 01:54:12.958700 18788 tablet_bootstrap.cc:492] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 311f765f59754bdcb4eadc04e14bd4ea: Bootstrap starting.
I20240502 01:54:12.959538 18785 tablet_bootstrap.cc:492] T 3bdd49939e994240a6d15dbd11559ff4 P da3584f429104d23a1a8899b35cc2138: Bootstrap starting.
I20240502 01:54:12.960995 18788 tablet_bootstrap.cc:654] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 311f765f59754bdcb4eadc04e14bd4ea: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:12.961735 18785 tablet_bootstrap.cc:654] T 3bdd49939e994240a6d15dbd11559ff4 P da3584f429104d23a1a8899b35cc2138: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:12.965126 18788 tablet_bootstrap.cc:492] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 311f765f59754bdcb4eadc04e14bd4ea: No bootstrap required, opened a new log
I20240502 01:54:12.965389 18788 ts_tablet_manager.cc:1367] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 311f765f59754bdcb4eadc04e14bd4ea: Time spent bootstrapping tablet: real 0.007s	user 0.006s	sys 0.000s
I20240502 01:54:12.965549 18785 tablet_bootstrap.cc:492] T 3bdd49939e994240a6d15dbd11559ff4 P da3584f429104d23a1a8899b35cc2138: No bootstrap required, opened a new log
I20240502 01:54:12.965847 18785 ts_tablet_manager.cc:1367] T 3bdd49939e994240a6d15dbd11559ff4 P da3584f429104d23a1a8899b35cc2138: Time spent bootstrapping tablet: real 0.006s	user 0.000s	sys 0.006s
I20240502 01:54:12.966367 18788 raft_consensus.cc:348] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } }
I20240502 01:54:12.966687 18788 raft_consensus.cc:374] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:12.966871 18788 raft_consensus.cc:729] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 311f765f59754bdcb4eadc04e14bd4ea, State: Initialized, Role: FOLLOWER
I20240502 01:54:12.966933 18787 tablet_bootstrap.cc:492] T 3bdd49939e994240a6d15dbd11559ff4 P 84d3a4b174c34e02aedab1e620731359: Bootstrap starting.
I20240502 01:54:12.967262 18788 consensus_queue.cc:260] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 311f765f59754bdcb4eadc04e14bd4ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } }
I20240502 01:54:12.967260 18785 raft_consensus.cc:348] T 3bdd49939e994240a6d15dbd11559ff4 P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } }
I20240502 01:54:12.967710 18785 raft_consensus.cc:374] T 3bdd49939e994240a6d15dbd11559ff4 P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:12.967875 18785 raft_consensus.cc:729] T 3bdd49939e994240a6d15dbd11559ff4 P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da3584f429104d23a1a8899b35cc2138, State: Initialized, Role: FOLLOWER
I20240502 01:54:12.968106 18788 ts_tablet_manager.cc:1398] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 311f765f59754bdcb4eadc04e14bd4ea: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 01:54:12.968302 18785 consensus_queue.cc:260] T 3bdd49939e994240a6d15dbd11559ff4 P da3584f429104d23a1a8899b35cc2138 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } }
I20240502 01:54:12.969609 18787 tablet_bootstrap.cc:654] T 3bdd49939e994240a6d15dbd11559ff4 P 84d3a4b174c34e02aedab1e620731359: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:12.971122 18784 tablet_bootstrap.cc:492] T 88a137bcfc76410583f910b377d1be7f P 1b50a031f2784ba498b3e52e16916205: Bootstrap starting.
I20240502 01:54:12.972023 18785 ts_tablet_manager.cc:1398] T 3bdd49939e994240a6d15dbd11559ff4 P da3584f429104d23a1a8899b35cc2138: Time spent starting tablet: real 0.006s	user 0.001s	sys 0.002s
I20240502 01:54:12.973001 18788 tablet_bootstrap.cc:492] T 88a137bcfc76410583f910b377d1be7f P 311f765f59754bdcb4eadc04e14bd4ea: Bootstrap starting.
I20240502 01:54:12.974066 18787 tablet_bootstrap.cc:492] T 3bdd49939e994240a6d15dbd11559ff4 P 84d3a4b174c34e02aedab1e620731359: No bootstrap required, opened a new log
I20240502 01:54:12.974242 18784 tablet_bootstrap.cc:654] T 88a137bcfc76410583f910b377d1be7f P 1b50a031f2784ba498b3e52e16916205: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:12.974301 18787 ts_tablet_manager.cc:1367] T 3bdd49939e994240a6d15dbd11559ff4 P 84d3a4b174c34e02aedab1e620731359: Time spent bootstrapping tablet: real 0.008s	user 0.004s	sys 0.000s
I20240502 01:54:12.974803 18788 tablet_bootstrap.cc:654] T 88a137bcfc76410583f910b377d1be7f P 311f765f59754bdcb4eadc04e14bd4ea: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:12.975433 18787 raft_consensus.cc:348] T 3bdd49939e994240a6d15dbd11559ff4 P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } }
I20240502 01:54:12.975745 18787 raft_consensus.cc:374] T 3bdd49939e994240a6d15dbd11559ff4 P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:12.975905 18787 raft_consensus.cc:729] T 3bdd49939e994240a6d15dbd11559ff4 P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84d3a4b174c34e02aedab1e620731359, State: Initialized, Role: FOLLOWER
I20240502 01:54:12.976245 18787 consensus_queue.cc:260] T 3bdd49939e994240a6d15dbd11559ff4 P 84d3a4b174c34e02aedab1e620731359 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } }
I20240502 01:54:12.976693 18784 tablet_bootstrap.cc:492] T 88a137bcfc76410583f910b377d1be7f P 1b50a031f2784ba498b3e52e16916205: No bootstrap required, opened a new log
I20240502 01:54:12.976814 18788 tablet_bootstrap.cc:492] T 88a137bcfc76410583f910b377d1be7f P 311f765f59754bdcb4eadc04e14bd4ea: No bootstrap required, opened a new log
I20240502 01:54:12.976868 18784 ts_tablet_manager.cc:1367] T 88a137bcfc76410583f910b377d1be7f P 1b50a031f2784ba498b3e52e16916205: Time spent bootstrapping tablet: real 0.006s	user 0.004s	sys 0.000s
I20240502 01:54:12.976948 18787 ts_tablet_manager.cc:1398] T 3bdd49939e994240a6d15dbd11559ff4 P 84d3a4b174c34e02aedab1e620731359: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 01:54:12.976979 18788 ts_tablet_manager.cc:1367] T 88a137bcfc76410583f910b377d1be7f P 311f765f59754bdcb4eadc04e14bd4ea: Time spent bootstrapping tablet: real 0.004s	user 0.002s	sys 0.001s
I20240502 01:54:12.977823 18784 raft_consensus.cc:348] T 88a137bcfc76410583f910b377d1be7f P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } }
I20240502 01:54:12.978047 18784 raft_consensus.cc:374] T 88a137bcfc76410583f910b377d1be7f P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:12.978140 18784 raft_consensus.cc:729] T 88a137bcfc76410583f910b377d1be7f P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b50a031f2784ba498b3e52e16916205, State: Initialized, Role: FOLLOWER
I20240502 01:54:12.978420 18787 tablet_bootstrap.cc:492] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 84d3a4b174c34e02aedab1e620731359: Bootstrap starting.
I20240502 01:54:12.978410 18784 consensus_queue.cc:260] T 88a137bcfc76410583f910b377d1be7f P 1b50a031f2784ba498b3e52e16916205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } }
I20240502 01:54:12.979089 18784 ts_tablet_manager.cc:1398] T 88a137bcfc76410583f910b377d1be7f P 1b50a031f2784ba498b3e52e16916205: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 01:54:12.979369 18784 tablet_bootstrap.cc:492] T 3bdd49939e994240a6d15dbd11559ff4 P 1b50a031f2784ba498b3e52e16916205: Bootstrap starting.
I20240502 01:54:12.979737 18788 raft_consensus.cc:348] T 88a137bcfc76410583f910b377d1be7f P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } }
I20240502 01:54:12.979974 18788 raft_consensus.cc:374] T 88a137bcfc76410583f910b377d1be7f P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:12.980078 18788 raft_consensus.cc:729] T 88a137bcfc76410583f910b377d1be7f P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 311f765f59754bdcb4eadc04e14bd4ea, State: Initialized, Role: FOLLOWER
I20240502 01:54:12.980384 18788 consensus_queue.cc:260] T 88a137bcfc76410583f910b377d1be7f P 311f765f59754bdcb4eadc04e14bd4ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } }
I20240502 01:54:12.980662 18787 tablet_bootstrap.cc:654] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 84d3a4b174c34e02aedab1e620731359: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:12.981824 18788 ts_tablet_manager.cc:1398] T 88a137bcfc76410583f910b377d1be7f P 311f765f59754bdcb4eadc04e14bd4ea: Time spent starting tablet: real 0.005s	user 0.001s	sys 0.002s
I20240502 01:54:12.982473 18784 tablet_bootstrap.cc:654] T 3bdd49939e994240a6d15dbd11559ff4 P 1b50a031f2784ba498b3e52e16916205: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:12.984349 18784 tablet_bootstrap.cc:492] T 3bdd49939e994240a6d15dbd11559ff4 P 1b50a031f2784ba498b3e52e16916205: No bootstrap required, opened a new log
I20240502 01:54:12.984567 18784 ts_tablet_manager.cc:1367] T 3bdd49939e994240a6d15dbd11559ff4 P 1b50a031f2784ba498b3e52e16916205: Time spent bootstrapping tablet: real 0.005s	user 0.003s	sys 0.000s
I20240502 01:54:12.985438 18784 raft_consensus.cc:348] T 3bdd49939e994240a6d15dbd11559ff4 P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } }
I20240502 01:54:12.985637 18784 raft_consensus.cc:374] T 3bdd49939e994240a6d15dbd11559ff4 P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:12.985725 18784 raft_consensus.cc:729] T 3bdd49939e994240a6d15dbd11559ff4 P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b50a031f2784ba498b3e52e16916205, State: Initialized, Role: FOLLOWER
I20240502 01:54:12.985991 18784 consensus_queue.cc:260] T 3bdd49939e994240a6d15dbd11559ff4 P 1b50a031f2784ba498b3e52e16916205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } }
I20240502 01:54:12.986569 18784 ts_tablet_manager.cc:1398] T 3bdd49939e994240a6d15dbd11559ff4 P 1b50a031f2784ba498b3e52e16916205: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240502 01:54:12.992420 18787 tablet_bootstrap.cc:492] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 84d3a4b174c34e02aedab1e620731359: No bootstrap required, opened a new log
I20240502 01:54:12.992637 18787 ts_tablet_manager.cc:1367] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 84d3a4b174c34e02aedab1e620731359: Time spent bootstrapping tablet: real 0.014s	user 0.005s	sys 0.008s
I20240502 01:54:12.993527 18787 raft_consensus.cc:348] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } }
I20240502 01:54:12.993718 18787 raft_consensus.cc:374] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:12.993781 18787 raft_consensus.cc:729] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84d3a4b174c34e02aedab1e620731359, State: Initialized, Role: FOLLOWER
I20240502 01:54:12.994011 18787 consensus_queue.cc:260] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 84d3a4b174c34e02aedab1e620731359 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } }
I20240502 01:54:12.994519 18787 ts_tablet_manager.cc:1398] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 84d3a4b174c34e02aedab1e620731359: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 01:54:12.994845 18787 tablet_bootstrap.cc:492] T 88a137bcfc76410583f910b377d1be7f P 84d3a4b174c34e02aedab1e620731359: Bootstrap starting.
I20240502 01:54:12.996507 18787 tablet_bootstrap.cc:654] T 88a137bcfc76410583f910b377d1be7f P 84d3a4b174c34e02aedab1e620731359: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:12.998394 18787 tablet_bootstrap.cc:492] T 88a137bcfc76410583f910b377d1be7f P 84d3a4b174c34e02aedab1e620731359: No bootstrap required, opened a new log
I20240502 01:54:12.998570 18787 ts_tablet_manager.cc:1367] T 88a137bcfc76410583f910b377d1be7f P 84d3a4b174c34e02aedab1e620731359: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.000s
I20240502 01:54:12.999307 18787 raft_consensus.cc:348] T 88a137bcfc76410583f910b377d1be7f P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } }
I20240502 01:54:12.999477 18787 raft_consensus.cc:374] T 88a137bcfc76410583f910b377d1be7f P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:12.999615 18787 raft_consensus.cc:729] T 88a137bcfc76410583f910b377d1be7f P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84d3a4b174c34e02aedab1e620731359, State: Initialized, Role: FOLLOWER
I20240502 01:54:12.999825 18787 consensus_queue.cc:260] T 88a137bcfc76410583f910b377d1be7f P 84d3a4b174c34e02aedab1e620731359 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } }
I20240502 01:54:13.000298 18787 ts_tablet_manager.cc:1398] T 88a137bcfc76410583f910b377d1be7f P 84d3a4b174c34e02aedab1e620731359: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240502 01:54:13.017520 18677 raft_consensus.cc:482] T 88a137bcfc76410583f910b377d1be7f P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:13.017746 18677 raft_consensus.cc:504] T 88a137bcfc76410583f910b377d1be7f P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } }
I20240502 01:54:13.018317 18677 leader_election.cc:288] T 88a137bcfc76410583f910b377d1be7f P 84d3a4b174c34e02aedab1e620731359 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193:35397), 1b50a031f2784ba498b3e52e16916205 (127.9.176.194:45617)
I20240502 01:54:13.018940 17498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88a137bcfc76410583f910b377d1be7f" candidate_uuid: "84d3a4b174c34e02aedab1e620731359" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "311f765f59754bdcb4eadc04e14bd4ea" is_pre_election: true
I20240502 01:54:13.018950 17633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88a137bcfc76410583f910b377d1be7f" candidate_uuid: "84d3a4b174c34e02aedab1e620731359" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b50a031f2784ba498b3e52e16916205" is_pre_election: true
I20240502 01:54:13.019148 17498 raft_consensus.cc:2454] T 88a137bcfc76410583f910b377d1be7f P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84d3a4b174c34e02aedab1e620731359 in term 0.
I20240502 01:54:13.019203 17633 raft_consensus.cc:2454] T 88a137bcfc76410583f910b377d1be7f P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84d3a4b174c34e02aedab1e620731359 in term 0.
I20240502 01:54:13.019707 17703 leader_election.cc:302] T 88a137bcfc76410583f910b377d1be7f P 84d3a4b174c34e02aedab1e620731359 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 311f765f59754bdcb4eadc04e14bd4ea, 84d3a4b174c34e02aedab1e620731359; no voters: 
I20240502 01:54:13.020015 18677 raft_consensus.cc:2781] T 88a137bcfc76410583f910b377d1be7f P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:13.020108 18677 raft_consensus.cc:482] T 88a137bcfc76410583f910b377d1be7f P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:13.020221 18677 raft_consensus.cc:3037] T 88a137bcfc76410583f910b377d1be7f P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:13.022122 18677 raft_consensus.cc:504] T 88a137bcfc76410583f910b377d1be7f P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } }
I20240502 01:54:13.022781 18677 leader_election.cc:288] T 88a137bcfc76410583f910b377d1be7f P 84d3a4b174c34e02aedab1e620731359 [CANDIDATE]: Term 1 election: Requested vote from peers 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193:35397), 1b50a031f2784ba498b3e52e16916205 (127.9.176.194:45617)
I20240502 01:54:13.023373 17498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88a137bcfc76410583f910b377d1be7f" candidate_uuid: "84d3a4b174c34e02aedab1e620731359" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "311f765f59754bdcb4eadc04e14bd4ea"
I20240502 01:54:13.023453 17633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88a137bcfc76410583f910b377d1be7f" candidate_uuid: "84d3a4b174c34e02aedab1e620731359" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b50a031f2784ba498b3e52e16916205"
I20240502 01:54:13.023644 17498 raft_consensus.cc:3037] T 88a137bcfc76410583f910b377d1be7f P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:13.023710 17633 raft_consensus.cc:3037] T 88a137bcfc76410583f910b377d1be7f P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:13.026535 17633 raft_consensus.cc:2454] T 88a137bcfc76410583f910b377d1be7f P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84d3a4b174c34e02aedab1e620731359 in term 1.
I20240502 01:54:13.026638 17498 raft_consensus.cc:2454] T 88a137bcfc76410583f910b377d1be7f P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84d3a4b174c34e02aedab1e620731359 in term 1.
I20240502 01:54:13.027029 17703 leader_election.cc:302] T 88a137bcfc76410583f910b377d1be7f P 84d3a4b174c34e02aedab1e620731359 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b50a031f2784ba498b3e52e16916205, 84d3a4b174c34e02aedab1e620731359; no voters: 
I20240502 01:54:13.027333 18677 raft_consensus.cc:2781] T 88a137bcfc76410583f910b377d1be7f P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:13.027529 18677 raft_consensus.cc:686] T 88a137bcfc76410583f910b377d1be7f P 84d3a4b174c34e02aedab1e620731359 [term 1 LEADER]: Becoming Leader. State: Replica: 84d3a4b174c34e02aedab1e620731359, State: Running, Role: LEADER
I20240502 01:54:13.027813 18677 consensus_queue.cc:237] T 88a137bcfc76410583f910b377d1be7f P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } }
I20240502 01:54:13.030717 17353 catalog_manager.cc:5484] T 88a137bcfc76410583f910b377d1be7f P 84d3a4b174c34e02aedab1e620731359 reported cstate change: term changed from 0 to 1, leader changed from <none> to 84d3a4b174c34e02aedab1e620731359 (127.9.176.195). New cstate: current_term: 1 leader_uuid: "84d3a4b174c34e02aedab1e620731359" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:13.048856 18677 raft_consensus.cc:482] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:13.049125 18677 raft_consensus.cc:504] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } }
I20240502 01:54:13.049933 18677 leader_election.cc:288] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 84d3a4b174c34e02aedab1e620731359 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers da3584f429104d23a1a8899b35cc2138 (127.9.176.196:46381), 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193:35397)
I20240502 01:54:13.050452 17899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ca5c4988ff74e3ca8d7a5614bcb93e2" candidate_uuid: "84d3a4b174c34e02aedab1e620731359" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da3584f429104d23a1a8899b35cc2138" is_pre_election: true
I20240502 01:54:13.050520 17498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ca5c4988ff74e3ca8d7a5614bcb93e2" candidate_uuid: "84d3a4b174c34e02aedab1e620731359" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "311f765f59754bdcb4eadc04e14bd4ea" is_pre_election: true
I20240502 01:54:13.050693 17899 raft_consensus.cc:2454] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84d3a4b174c34e02aedab1e620731359 in term 0.
I20240502 01:54:13.050810 17498 raft_consensus.cc:2454] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84d3a4b174c34e02aedab1e620731359 in term 0.
I20240502 01:54:13.051159 17700 leader_election.cc:302] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 84d3a4b174c34e02aedab1e620731359 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84d3a4b174c34e02aedab1e620731359, da3584f429104d23a1a8899b35cc2138; no voters: 
I20240502 01:54:13.051458 18677 raft_consensus.cc:2781] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:13.051555 18677 raft_consensus.cc:482] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:13.051673 18677 raft_consensus.cc:3037] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:13.053532 18677 raft_consensus.cc:504] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } }
I20240502 01:54:13.054203 18677 leader_election.cc:288] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 84d3a4b174c34e02aedab1e620731359 [CANDIDATE]: Term 1 election: Requested vote from peers da3584f429104d23a1a8899b35cc2138 (127.9.176.196:46381), 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193:35397)
I20240502 01:54:13.054731 17899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ca5c4988ff74e3ca8d7a5614bcb93e2" candidate_uuid: "84d3a4b174c34e02aedab1e620731359" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da3584f429104d23a1a8899b35cc2138"
I20240502 01:54:13.054720 17498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ca5c4988ff74e3ca8d7a5614bcb93e2" candidate_uuid: "84d3a4b174c34e02aedab1e620731359" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "311f765f59754bdcb4eadc04e14bd4ea"
I20240502 01:54:13.054939 17899 raft_consensus.cc:3037] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:13.054955 17498 raft_consensus.cc:3037] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:13.057039 17498 raft_consensus.cc:2454] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84d3a4b174c34e02aedab1e620731359 in term 1.
I20240502 01:54:13.057056 17899 raft_consensus.cc:2454] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84d3a4b174c34e02aedab1e620731359 in term 1.
I20240502 01:54:13.057555 17700 leader_election.cc:302] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 84d3a4b174c34e02aedab1e620731359 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84d3a4b174c34e02aedab1e620731359, da3584f429104d23a1a8899b35cc2138; no voters: 
I20240502 01:54:13.057919 18677 raft_consensus.cc:2781] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:13.058081 18677 raft_consensus.cc:686] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 84d3a4b174c34e02aedab1e620731359 [term 1 LEADER]: Becoming Leader. State: Replica: 84d3a4b174c34e02aedab1e620731359, State: Running, Role: LEADER
I20240502 01:54:13.058393 18677 consensus_queue.cc:237] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } }
I20240502 01:54:13.061339 17353 catalog_manager.cc:5484] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 84d3a4b174c34e02aedab1e620731359 reported cstate change: term changed from 0 to 1, leader changed from <none> to 84d3a4b174c34e02aedab1e620731359 (127.9.176.195). New cstate: current_term: 1 leader_uuid: "84d3a4b174c34e02aedab1e620731359" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:13.298439 18657 raft_consensus.cc:482] T 3bdd49939e994240a6d15dbd11559ff4 P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:13.298646 18657 raft_consensus.cc:504] T 3bdd49939e994240a6d15dbd11559ff4 P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } }
I20240502 01:54:13.300222 18657 leader_election.cc:288] T 3bdd49939e994240a6d15dbd11559ff4 P da3584f429104d23a1a8899b35cc2138 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b50a031f2784ba498b3e52e16916205 (127.9.176.194:45617), 84d3a4b174c34e02aedab1e620731359 (127.9.176.195:37357)
I20240502 01:54:13.308832 17767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bdd49939e994240a6d15dbd11559ff4" candidate_uuid: "da3584f429104d23a1a8899b35cc2138" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84d3a4b174c34e02aedab1e620731359" is_pre_election: true
I20240502 01:54:13.308905 17633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bdd49939e994240a6d15dbd11559ff4" candidate_uuid: "da3584f429104d23a1a8899b35cc2138" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b50a031f2784ba498b3e52e16916205" is_pre_election: true
I20240502 01:54:13.309149 17767 raft_consensus.cc:2454] T 3bdd49939e994240a6d15dbd11559ff4 P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da3584f429104d23a1a8899b35cc2138 in term 0.
I20240502 01:54:13.309149 17633 raft_consensus.cc:2454] T 3bdd49939e994240a6d15dbd11559ff4 P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da3584f429104d23a1a8899b35cc2138 in term 0.
I20240502 01:54:13.309677 17835 leader_election.cc:302] T 3bdd49939e994240a6d15dbd11559ff4 P da3584f429104d23a1a8899b35cc2138 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84d3a4b174c34e02aedab1e620731359, da3584f429104d23a1a8899b35cc2138; no voters: 
I20240502 01:54:13.310050 18657 raft_consensus.cc:2781] T 3bdd49939e994240a6d15dbd11559ff4 P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:13.310182 18657 raft_consensus.cc:482] T 3bdd49939e994240a6d15dbd11559ff4 P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:13.310297 18657 raft_consensus.cc:3037] T 3bdd49939e994240a6d15dbd11559ff4 P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:13.312433 18657 raft_consensus.cc:504] T 3bdd49939e994240a6d15dbd11559ff4 P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } }
I20240502 01:54:13.313125 18657 leader_election.cc:288] T 3bdd49939e994240a6d15dbd11559ff4 P da3584f429104d23a1a8899b35cc2138 [CANDIDATE]: Term 1 election: Requested vote from peers 1b50a031f2784ba498b3e52e16916205 (127.9.176.194:45617), 84d3a4b174c34e02aedab1e620731359 (127.9.176.195:37357)
I20240502 01:54:13.313709 17633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bdd49939e994240a6d15dbd11559ff4" candidate_uuid: "da3584f429104d23a1a8899b35cc2138" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b50a031f2784ba498b3e52e16916205"
I20240502 01:54:13.313774 17767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bdd49939e994240a6d15dbd11559ff4" candidate_uuid: "da3584f429104d23a1a8899b35cc2138" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84d3a4b174c34e02aedab1e620731359"
I20240502 01:54:13.313966 17633 raft_consensus.cc:3037] T 3bdd49939e994240a6d15dbd11559ff4 P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:13.313977 17767 raft_consensus.cc:3037] T 3bdd49939e994240a6d15dbd11559ff4 P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:13.316746 17767 raft_consensus.cc:2454] T 3bdd49939e994240a6d15dbd11559ff4 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da3584f429104d23a1a8899b35cc2138 in term 1.
I20240502 01:54:13.317194 17835 leader_election.cc:302] T 3bdd49939e994240a6d15dbd11559ff4 P da3584f429104d23a1a8899b35cc2138 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84d3a4b174c34e02aedab1e620731359, da3584f429104d23a1a8899b35cc2138; no voters: 
I20240502 01:54:13.317250 17633 raft_consensus.cc:2454] T 3bdd49939e994240a6d15dbd11559ff4 P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da3584f429104d23a1a8899b35cc2138 in term 1.
I20240502 01:54:13.317584 18657 raft_consensus.cc:2781] T 3bdd49939e994240a6d15dbd11559ff4 P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:13.317987 18657 raft_consensus.cc:686] T 3bdd49939e994240a6d15dbd11559ff4 P da3584f429104d23a1a8899b35cc2138 [term 1 LEADER]: Becoming Leader. State: Replica: da3584f429104d23a1a8899b35cc2138, State: Running, Role: LEADER
I20240502 01:54:13.318486 18657 consensus_queue.cc:237] T 3bdd49939e994240a6d15dbd11559ff4 P da3584f429104d23a1a8899b35cc2138 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } }
I20240502 01:54:13.322253 17353 catalog_manager.cc:5484] T 3bdd49939e994240a6d15dbd11559ff4 P da3584f429104d23a1a8899b35cc2138 reported cstate change: term changed from 0 to 1, leader changed from <none> to da3584f429104d23a1a8899b35cc2138 (127.9.176.196). New cstate: current_term: 1 leader_uuid: "da3584f429104d23a1a8899b35cc2138" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:13.466965 18677 consensus_queue.cc:1035] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:13.470460 18677 consensus_queue.cc:1035] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:13.488466 18677 consensus_queue.cc:1035] T 88a137bcfc76410583f910b377d1be7f P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:13.498390 18810 consensus_queue.cc:1035] T 88a137bcfc76410583f910b377d1be7f P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:54:13.785331 18805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:54:13.785691 18805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:54:13.793771 18805 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 01:54:13.819095 18657 consensus_queue.cc:1035] T 3bdd49939e994240a6d15dbd11559ff4 P da3584f429104d23a1a8899b35cc2138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:13.825675 18829 consensus_queue.cc:1035] T 3bdd49939e994240a6d15dbd11559ff4 P da3584f429104d23a1a8899b35cc2138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_3'

INSERT report
    rows total: 2000
    time total: 260.474 ms
  time per row: 0.130237 ms
I20240502 01:54:14.212239 17349 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56698:
name: "rebalance_test_table_4"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 01:54:14.223965 17477 tablet_service.cc:1467] Processing CreateTablet for tablet f07e9b433e554c85b3c25b7e708f8b2f (DEFAULT_TABLE table=rebalance_test_table_4 [id=14b25b3f4ede4b65870815b014d90449]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:14.224244 17611 tablet_service.cc:1467] Processing CreateTablet for tablet 59e507855e5043d38cef84e92adf437b (DEFAULT_TABLE table=rebalance_test_table_4 [id=14b25b3f4ede4b65870815b014d90449]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:14.224526 17477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f07e9b433e554c85b3c25b7e708f8b2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:14.224615 17612 tablet_service.cc:1467] Processing CreateTablet for tablet f07e9b433e554c85b3c25b7e708f8b2f (DEFAULT_TABLE table=rebalance_test_table_4 [id=14b25b3f4ede4b65870815b014d90449]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:14.225339 17746 tablet_service.cc:1467] Processing CreateTablet for tablet f07e9b433e554c85b3c25b7e708f8b2f (DEFAULT_TABLE table=rebalance_test_table_4 [id=14b25b3f4ede4b65870815b014d90449]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:14.225751 17746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f07e9b433e554c85b3c25b7e708f8b2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:14.223948 17479 tablet_service.cc:1467] Processing CreateTablet for tablet 59e507855e5043d38cef84e92adf437b (DEFAULT_TABLE table=rebalance_test_table_4 [id=14b25b3f4ede4b65870815b014d90449]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:14.227784 17479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59e507855e5043d38cef84e92adf437b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:14.231374 17878 tablet_service.cc:1467] Processing CreateTablet for tablet 59e507855e5043d38cef84e92adf437b (DEFAULT_TABLE table=rebalance_test_table_4 [id=14b25b3f4ede4b65870815b014d90449]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:14.231521 17880 tablet_service.cc:1467] Processing CreateTablet for tablet a5e03cf44eab41cc842342a138c04067 (DEFAULT_TABLE table=rebalance_test_table_4 [id=14b25b3f4ede4b65870815b014d90449]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:14.231979 17880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5e03cf44eab41cc842342a138c04067. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:14.233126 17878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59e507855e5043d38cef84e92adf437b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:14.240518 17747 tablet_service.cc:1467] Processing CreateTablet for tablet a5e03cf44eab41cc842342a138c04067 (DEFAULT_TABLE table=rebalance_test_table_4 [id=14b25b3f4ede4b65870815b014d90449]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:14.240671 17478 tablet_service.cc:1467] Processing CreateTablet for tablet a5e03cf44eab41cc842342a138c04067 (DEFAULT_TABLE table=rebalance_test_table_4 [id=14b25b3f4ede4b65870815b014d90449]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:14.240900 18868 tablet_bootstrap.cc:492] T f07e9b433e554c85b3c25b7e708f8b2f P 84d3a4b174c34e02aedab1e620731359: Bootstrap starting.
I20240502 01:54:14.240970 17747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5e03cf44eab41cc842342a138c04067. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:14.241147 17478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5e03cf44eab41cc842342a138c04067. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:14.224730 17611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59e507855e5043d38cef84e92adf437b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:14.246265 17612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f07e9b433e554c85b3c25b7e708f8b2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:14.256587 18869 tablet_bootstrap.cc:492] T a5e03cf44eab41cc842342a138c04067 P 311f765f59754bdcb4eadc04e14bd4ea: Bootstrap starting.
I20240502 01:54:14.258209 18870 tablet_bootstrap.cc:492] T 59e507855e5043d38cef84e92adf437b P da3584f429104d23a1a8899b35cc2138: Bootstrap starting.
I20240502 01:54:14.258582 18869 tablet_bootstrap.cc:654] T a5e03cf44eab41cc842342a138c04067 P 311f765f59754bdcb4eadc04e14bd4ea: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:14.260651 18870 tablet_bootstrap.cc:654] T 59e507855e5043d38cef84e92adf437b P da3584f429104d23a1a8899b35cc2138: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:14.261229 18868 tablet_bootstrap.cc:654] T f07e9b433e554c85b3c25b7e708f8b2f P 84d3a4b174c34e02aedab1e620731359: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:14.261937 18871 tablet_bootstrap.cc:492] T f07e9b433e554c85b3c25b7e708f8b2f P 1b50a031f2784ba498b3e52e16916205: Bootstrap starting.
I20240502 01:54:14.265156 18871 tablet_bootstrap.cc:654] T f07e9b433e554c85b3c25b7e708f8b2f P 1b50a031f2784ba498b3e52e16916205: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:14.267742 18868 tablet_bootstrap.cc:492] T f07e9b433e554c85b3c25b7e708f8b2f P 84d3a4b174c34e02aedab1e620731359: No bootstrap required, opened a new log
I20240502 01:54:14.267977 18868 ts_tablet_manager.cc:1367] T f07e9b433e554c85b3c25b7e708f8b2f P 84d3a4b174c34e02aedab1e620731359: Time spent bootstrapping tablet: real 0.027s	user 0.004s	sys 0.002s
I20240502 01:54:14.269121 18868 raft_consensus.cc:348] T f07e9b433e554c85b3c25b7e708f8b2f P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } }
I20240502 01:54:14.269436 18868 raft_consensus.cc:374] T f07e9b433e554c85b3c25b7e708f8b2f P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:14.269558 18868 raft_consensus.cc:729] T f07e9b433e554c85b3c25b7e708f8b2f P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84d3a4b174c34e02aedab1e620731359, State: Initialized, Role: FOLLOWER
I20240502 01:54:14.269938 18868 consensus_queue.cc:260] T f07e9b433e554c85b3c25b7e708f8b2f P 84d3a4b174c34e02aedab1e620731359 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } }
I20240502 01:54:14.270785 18868 ts_tablet_manager.cc:1398] T f07e9b433e554c85b3c25b7e708f8b2f P 84d3a4b174c34e02aedab1e620731359: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.001s
I20240502 01:54:14.271081 18868 tablet_bootstrap.cc:492] T a5e03cf44eab41cc842342a138c04067 P 84d3a4b174c34e02aedab1e620731359: Bootstrap starting.
I20240502 01:54:14.275102 18868 tablet_bootstrap.cc:654] T a5e03cf44eab41cc842342a138c04067 P 84d3a4b174c34e02aedab1e620731359: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:14.275903 18871 tablet_bootstrap.cc:492] T f07e9b433e554c85b3c25b7e708f8b2f P 1b50a031f2784ba498b3e52e16916205: No bootstrap required, opened a new log
I20240502 01:54:14.276139 18869 tablet_bootstrap.cc:492] T a5e03cf44eab41cc842342a138c04067 P 311f765f59754bdcb4eadc04e14bd4ea: No bootstrap required, opened a new log
I20240502 01:54:14.276134 18871 ts_tablet_manager.cc:1367] T f07e9b433e554c85b3c25b7e708f8b2f P 1b50a031f2784ba498b3e52e16916205: Time spent bootstrapping tablet: real 0.014s	user 0.008s	sys 0.001s
I20240502 01:54:14.276360 18869 ts_tablet_manager.cc:1367] T a5e03cf44eab41cc842342a138c04067 P 311f765f59754bdcb4eadc04e14bd4ea: Time spent bootstrapping tablet: real 0.021s	user 0.009s	sys 0.008s
I20240502 01:54:14.277266 18869 raft_consensus.cc:348] T a5e03cf44eab41cc842342a138c04067 P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } }
I20240502 01:54:14.277555 18869 raft_consensus.cc:374] T a5e03cf44eab41cc842342a138c04067 P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:14.277694 18869 raft_consensus.cc:729] T a5e03cf44eab41cc842342a138c04067 P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 311f765f59754bdcb4eadc04e14bd4ea, State: Initialized, Role: FOLLOWER
I20240502 01:54:14.277581 18871 raft_consensus.cc:348] T f07e9b433e554c85b3c25b7e708f8b2f P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } }
I20240502 01:54:14.277910 18871 raft_consensus.cc:374] T f07e9b433e554c85b3c25b7e708f8b2f P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:14.278043 18871 raft_consensus.cc:729] T f07e9b433e554c85b3c25b7e708f8b2f P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b50a031f2784ba498b3e52e16916205, State: Initialized, Role: FOLLOWER
I20240502 01:54:14.278055 18869 consensus_queue.cc:260] T a5e03cf44eab41cc842342a138c04067 P 311f765f59754bdcb4eadc04e14bd4ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } }
I20240502 01:54:14.278322 18871 consensus_queue.cc:260] T f07e9b433e554c85b3c25b7e708f8b2f P 1b50a031f2784ba498b3e52e16916205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } }
I20240502 01:54:14.278549 18870 tablet_bootstrap.cc:492] T 59e507855e5043d38cef84e92adf437b P da3584f429104d23a1a8899b35cc2138: No bootstrap required, opened a new log
I20240502 01:54:14.278882 18870 ts_tablet_manager.cc:1367] T 59e507855e5043d38cef84e92adf437b P da3584f429104d23a1a8899b35cc2138: Time spent bootstrapping tablet: real 0.021s	user 0.012s	sys 0.004s
I20240502 01:54:14.279022 18869 ts_tablet_manager.cc:1398] T a5e03cf44eab41cc842342a138c04067 P 311f765f59754bdcb4eadc04e14bd4ea: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 01:54:14.279057 18871 ts_tablet_manager.cc:1398] T f07e9b433e554c85b3c25b7e708f8b2f P 1b50a031f2784ba498b3e52e16916205: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.002s
I20240502 01:54:14.279392 18869 tablet_bootstrap.cc:492] T 59e507855e5043d38cef84e92adf437b P 311f765f59754bdcb4eadc04e14bd4ea: Bootstrap starting.
I20240502 01:54:14.279399 18871 tablet_bootstrap.cc:492] T 59e507855e5043d38cef84e92adf437b P 1b50a031f2784ba498b3e52e16916205: Bootstrap starting.
I20240502 01:54:14.280602 18870 raft_consensus.cc:348] T 59e507855e5043d38cef84e92adf437b P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } }
I20240502 01:54:14.280838 18870 raft_consensus.cc:374] T 59e507855e5043d38cef84e92adf437b P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:14.280948 18870 raft_consensus.cc:729] T 59e507855e5043d38cef84e92adf437b P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da3584f429104d23a1a8899b35cc2138, State: Initialized, Role: FOLLOWER
I20240502 01:54:14.281255 18871 tablet_bootstrap.cc:654] T 59e507855e5043d38cef84e92adf437b P 1b50a031f2784ba498b3e52e16916205: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:14.281234 18870 consensus_queue.cc:260] T 59e507855e5043d38cef84e92adf437b P da3584f429104d23a1a8899b35cc2138 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } }
I20240502 01:54:14.281988 18870 ts_tablet_manager.cc:1398] T 59e507855e5043d38cef84e92adf437b P da3584f429104d23a1a8899b35cc2138: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.000s
I20240502 01:54:14.282318 18870 tablet_bootstrap.cc:492] T a5e03cf44eab41cc842342a138c04067 P da3584f429104d23a1a8899b35cc2138: Bootstrap starting.
I20240502 01:54:14.283810 18869 tablet_bootstrap.cc:654] T 59e507855e5043d38cef84e92adf437b P 311f765f59754bdcb4eadc04e14bd4ea: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:14.285773 18870 tablet_bootstrap.cc:654] T a5e03cf44eab41cc842342a138c04067 P da3584f429104d23a1a8899b35cc2138: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:14.278504 18868 tablet_bootstrap.cc:492] T a5e03cf44eab41cc842342a138c04067 P 84d3a4b174c34e02aedab1e620731359: No bootstrap required, opened a new log
I20240502 01:54:14.286293 18868 ts_tablet_manager.cc:1367] T a5e03cf44eab41cc842342a138c04067 P 84d3a4b174c34e02aedab1e620731359: Time spent bootstrapping tablet: real 0.015s	user 0.005s	sys 0.001s
I20240502 01:54:14.287345 18868 raft_consensus.cc:348] T a5e03cf44eab41cc842342a138c04067 P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } }
I20240502 01:54:14.287681 18868 raft_consensus.cc:374] T a5e03cf44eab41cc842342a138c04067 P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:14.287786 18868 raft_consensus.cc:729] T a5e03cf44eab41cc842342a138c04067 P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84d3a4b174c34e02aedab1e620731359, State: Initialized, Role: FOLLOWER
I20240502 01:54:14.288172 18868 consensus_queue.cc:260] T a5e03cf44eab41cc842342a138c04067 P 84d3a4b174c34e02aedab1e620731359 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } }
I20240502 01:54:14.289608 18870 tablet_bootstrap.cc:492] T a5e03cf44eab41cc842342a138c04067 P da3584f429104d23a1a8899b35cc2138: No bootstrap required, opened a new log
I20240502 01:54:14.289795 18870 ts_tablet_manager.cc:1367] T a5e03cf44eab41cc842342a138c04067 P da3584f429104d23a1a8899b35cc2138: Time spent bootstrapping tablet: real 0.008s	user 0.001s	sys 0.003s
I20240502 01:54:14.290642 18870 raft_consensus.cc:348] T a5e03cf44eab41cc842342a138c04067 P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } }
I20240502 01:54:14.290903 18870 raft_consensus.cc:374] T a5e03cf44eab41cc842342a138c04067 P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:14.290995 18870 raft_consensus.cc:729] T a5e03cf44eab41cc842342a138c04067 P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da3584f429104d23a1a8899b35cc2138, State: Initialized, Role: FOLLOWER
I20240502 01:54:14.291290 18870 consensus_queue.cc:260] T a5e03cf44eab41cc842342a138c04067 P da3584f429104d23a1a8899b35cc2138 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } }
I20240502 01:54:14.292024 18870 ts_tablet_manager.cc:1398] T a5e03cf44eab41cc842342a138c04067 P da3584f429104d23a1a8899b35cc2138: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240502 01:54:14.293190 18868 ts_tablet_manager.cc:1398] T a5e03cf44eab41cc842342a138c04067 P 84d3a4b174c34e02aedab1e620731359: Time spent starting tablet: real 0.007s	user 0.002s	sys 0.000s
I20240502 01:54:14.294040 18871 tablet_bootstrap.cc:492] T 59e507855e5043d38cef84e92adf437b P 1b50a031f2784ba498b3e52e16916205: No bootstrap required, opened a new log
I20240502 01:54:14.294242 18871 ts_tablet_manager.cc:1367] T 59e507855e5043d38cef84e92adf437b P 1b50a031f2784ba498b3e52e16916205: Time spent bootstrapping tablet: real 0.015s	user 0.002s	sys 0.011s
I20240502 01:54:14.294984 18869 tablet_bootstrap.cc:492] T 59e507855e5043d38cef84e92adf437b P 311f765f59754bdcb4eadc04e14bd4ea: No bootstrap required, opened a new log
I20240502 01:54:14.295253 18869 ts_tablet_manager.cc:1367] T 59e507855e5043d38cef84e92adf437b P 311f765f59754bdcb4eadc04e14bd4ea: Time spent bootstrapping tablet: real 0.016s	user 0.003s	sys 0.010s
I20240502 01:54:14.295141 18871 raft_consensus.cc:348] T 59e507855e5043d38cef84e92adf437b P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } }
I20240502 01:54:14.295380 18871 raft_consensus.cc:374] T 59e507855e5043d38cef84e92adf437b P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:14.295469 18871 raft_consensus.cc:729] T 59e507855e5043d38cef84e92adf437b P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b50a031f2784ba498b3e52e16916205, State: Initialized, Role: FOLLOWER
I20240502 01:54:14.295791 18871 consensus_queue.cc:260] T 59e507855e5043d38cef84e92adf437b P 1b50a031f2784ba498b3e52e16916205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } }
I20240502 01:54:14.296554 18871 ts_tablet_manager.cc:1398] T 59e507855e5043d38cef84e92adf437b P 1b50a031f2784ba498b3e52e16916205: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.003s
I20240502 01:54:14.296439 18869 raft_consensus.cc:348] T 59e507855e5043d38cef84e92adf437b P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } }
I20240502 01:54:14.296751 18869 raft_consensus.cc:374] T 59e507855e5043d38cef84e92adf437b P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:14.296859 18869 raft_consensus.cc:729] T 59e507855e5043d38cef84e92adf437b P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 311f765f59754bdcb4eadc04e14bd4ea, State: Initialized, Role: FOLLOWER
I20240502 01:54:14.297147 18869 consensus_queue.cc:260] T 59e507855e5043d38cef84e92adf437b P 311f765f59754bdcb4eadc04e14bd4ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } }
I20240502 01:54:14.297627 18869 ts_tablet_manager.cc:1398] T 59e507855e5043d38cef84e92adf437b P 311f765f59754bdcb4eadc04e14bd4ea: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.001s
I20240502 01:54:14.297914 18869 tablet_bootstrap.cc:492] T f07e9b433e554c85b3c25b7e708f8b2f P 311f765f59754bdcb4eadc04e14bd4ea: Bootstrap starting.
I20240502 01:54:14.299607 18869 tablet_bootstrap.cc:654] T f07e9b433e554c85b3c25b7e708f8b2f P 311f765f59754bdcb4eadc04e14bd4ea: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:14.302573 18869 tablet_bootstrap.cc:492] T f07e9b433e554c85b3c25b7e708f8b2f P 311f765f59754bdcb4eadc04e14bd4ea: No bootstrap required, opened a new log
I20240502 01:54:14.302737 18869 ts_tablet_manager.cc:1367] T f07e9b433e554c85b3c25b7e708f8b2f P 311f765f59754bdcb4eadc04e14bd4ea: Time spent bootstrapping tablet: real 0.005s	user 0.003s	sys 0.000s
I20240502 01:54:14.303411 18869 raft_consensus.cc:348] T f07e9b433e554c85b3c25b7e708f8b2f P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } }
I20240502 01:54:14.303601 18828 raft_consensus.cc:482] T a5e03cf44eab41cc842342a138c04067 P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:14.303648 18869 raft_consensus.cc:374] T f07e9b433e554c85b3c25b7e708f8b2f P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:14.303740 18869 raft_consensus.cc:729] T f07e9b433e554c85b3c25b7e708f8b2f P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 311f765f59754bdcb4eadc04e14bd4ea, State: Initialized, Role: FOLLOWER
I20240502 01:54:14.303779 18828 raft_consensus.cc:504] T a5e03cf44eab41cc842342a138c04067 P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } }
I20240502 01:54:14.303953 18869 consensus_queue.cc:260] T f07e9b433e554c85b3c25b7e708f8b2f P 311f765f59754bdcb4eadc04e14bd4ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } }
I20240502 01:54:14.304527 18869 ts_tablet_manager.cc:1398] T f07e9b433e554c85b3c25b7e708f8b2f P 311f765f59754bdcb4eadc04e14bd4ea: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 01:54:14.304689 18828 leader_election.cc:288] T a5e03cf44eab41cc842342a138c04067 P da3584f429104d23a1a8899b35cc2138 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84d3a4b174c34e02aedab1e620731359 (127.9.176.195:37357), 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193:35397)
I20240502 01:54:14.305688 17767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5e03cf44eab41cc842342a138c04067" candidate_uuid: "da3584f429104d23a1a8899b35cc2138" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84d3a4b174c34e02aedab1e620731359" is_pre_election: true
I20240502 01:54:14.306018 17767 raft_consensus.cc:2454] T a5e03cf44eab41cc842342a138c04067 P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da3584f429104d23a1a8899b35cc2138 in term 0.
I20240502 01:54:14.306811 17835 leader_election.cc:302] T a5e03cf44eab41cc842342a138c04067 P da3584f429104d23a1a8899b35cc2138 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84d3a4b174c34e02aedab1e620731359, da3584f429104d23a1a8899b35cc2138; no voters: 
I20240502 01:54:14.307194 18828 raft_consensus.cc:2781] T a5e03cf44eab41cc842342a138c04067 P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:14.307314 18828 raft_consensus.cc:482] T a5e03cf44eab41cc842342a138c04067 P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:14.307420 18828 raft_consensus.cc:3037] T a5e03cf44eab41cc842342a138c04067 P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:14.309365 18828 raft_consensus.cc:504] T a5e03cf44eab41cc842342a138c04067 P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } }
I20240502 01:54:14.309935 18828 leader_election.cc:288] T a5e03cf44eab41cc842342a138c04067 P da3584f429104d23a1a8899b35cc2138 [CANDIDATE]: Term 1 election: Requested vote from peers 84d3a4b174c34e02aedab1e620731359 (127.9.176.195:37357), 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193:35397)
I20240502 01:54:14.310756 17767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5e03cf44eab41cc842342a138c04067" candidate_uuid: "da3584f429104d23a1a8899b35cc2138" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84d3a4b174c34e02aedab1e620731359"
I20240502 01:54:14.310977 17767 raft_consensus.cc:3037] T a5e03cf44eab41cc842342a138c04067 P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:14.313306 17767 raft_consensus.cc:2454] T a5e03cf44eab41cc842342a138c04067 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da3584f429104d23a1a8899b35cc2138 in term 1.
I20240502 01:54:14.313778 17835 leader_election.cc:302] T a5e03cf44eab41cc842342a138c04067 P da3584f429104d23a1a8899b35cc2138 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84d3a4b174c34e02aedab1e620731359, da3584f429104d23a1a8899b35cc2138; no voters: 
I20240502 01:54:14.314093 18828 raft_consensus.cc:2781] T a5e03cf44eab41cc842342a138c04067 P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:14.314384 18828 raft_consensus.cc:686] T a5e03cf44eab41cc842342a138c04067 P da3584f429104d23a1a8899b35cc2138 [term 1 LEADER]: Becoming Leader. State: Replica: da3584f429104d23a1a8899b35cc2138, State: Running, Role: LEADER
I20240502 01:54:14.314666 18828 consensus_queue.cc:237] T a5e03cf44eab41cc842342a138c04067 P da3584f429104d23a1a8899b35cc2138 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } }
I20240502 01:54:14.317816 17349 catalog_manager.cc:5484] T a5e03cf44eab41cc842342a138c04067 P da3584f429104d23a1a8899b35cc2138 reported cstate change: term changed from 0 to 1, leader changed from <none> to da3584f429104d23a1a8899b35cc2138 (127.9.176.196). New cstate: current_term: 1 leader_uuid: "da3584f429104d23a1a8899b35cc2138" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:14.318748 17498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5e03cf44eab41cc842342a138c04067" candidate_uuid: "da3584f429104d23a1a8899b35cc2138" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "311f765f59754bdcb4eadc04e14bd4ea" is_pre_election: true
I20240502 01:54:14.318779 17500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5e03cf44eab41cc842342a138c04067" candidate_uuid: "da3584f429104d23a1a8899b35cc2138" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "311f765f59754bdcb4eadc04e14bd4ea"
I20240502 01:54:14.319015 17498 raft_consensus.cc:2454] T a5e03cf44eab41cc842342a138c04067 P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da3584f429104d23a1a8899b35cc2138 in term 0.
I20240502 01:54:14.390671 18863 raft_consensus.cc:482] T 59e507855e5043d38cef84e92adf437b P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:14.390906 18863 raft_consensus.cc:504] T 59e507855e5043d38cef84e92adf437b P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } }
I20240502 01:54:14.391738 18863 leader_election.cc:288] T 59e507855e5043d38cef84e92adf437b P 311f765f59754bdcb4eadc04e14bd4ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b50a031f2784ba498b3e52e16916205 (127.9.176.194:45617), da3584f429104d23a1a8899b35cc2138 (127.9.176.196:46381)
I20240502 01:54:14.392334 17634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59e507855e5043d38cef84e92adf437b" candidate_uuid: "311f765f59754bdcb4eadc04e14bd4ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b50a031f2784ba498b3e52e16916205" is_pre_election: true
I20240502 01:54:14.392390 17899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59e507855e5043d38cef84e92adf437b" candidate_uuid: "311f765f59754bdcb4eadc04e14bd4ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da3584f429104d23a1a8899b35cc2138" is_pre_election: true
I20240502 01:54:14.392583 17634 raft_consensus.cc:2454] T 59e507855e5043d38cef84e92adf437b P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 311f765f59754bdcb4eadc04e14bd4ea in term 0.
I20240502 01:54:14.392697 17899 raft_consensus.cc:2454] T 59e507855e5043d38cef84e92adf437b P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 311f765f59754bdcb4eadc04e14bd4ea in term 0.
I20240502 01:54:14.393169 17435 leader_election.cc:302] T 59e507855e5043d38cef84e92adf437b P 311f765f59754bdcb4eadc04e14bd4ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1b50a031f2784ba498b3e52e16916205, 311f765f59754bdcb4eadc04e14bd4ea, da3584f429104d23a1a8899b35cc2138; no voters: 
I20240502 01:54:14.393513 18863 raft_consensus.cc:2781] T 59e507855e5043d38cef84e92adf437b P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:14.393633 18863 raft_consensus.cc:482] T 59e507855e5043d38cef84e92adf437b P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:14.393738 18863 raft_consensus.cc:3037] T 59e507855e5043d38cef84e92adf437b P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:14.395851 18863 raft_consensus.cc:504] T 59e507855e5043d38cef84e92adf437b P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } }
I20240502 01:54:14.396497 18863 leader_election.cc:288] T 59e507855e5043d38cef84e92adf437b P 311f765f59754bdcb4eadc04e14bd4ea [CANDIDATE]: Term 1 election: Requested vote from peers 1b50a031f2784ba498b3e52e16916205 (127.9.176.194:45617), da3584f429104d23a1a8899b35cc2138 (127.9.176.196:46381)
I20240502 01:54:14.397039 17899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59e507855e5043d38cef84e92adf437b" candidate_uuid: "311f765f59754bdcb4eadc04e14bd4ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da3584f429104d23a1a8899b35cc2138"
I20240502 01:54:14.397048 17634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59e507855e5043d38cef84e92adf437b" candidate_uuid: "311f765f59754bdcb4eadc04e14bd4ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b50a031f2784ba498b3e52e16916205"
I20240502 01:54:14.397244 17899 raft_consensus.cc:3037] T 59e507855e5043d38cef84e92adf437b P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:14.397262 17634 raft_consensus.cc:3037] T 59e507855e5043d38cef84e92adf437b P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:14.399138 17634 raft_consensus.cc:2454] T 59e507855e5043d38cef84e92adf437b P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 311f765f59754bdcb4eadc04e14bd4ea in term 1.
I20240502 01:54:14.399149 17899 raft_consensus.cc:2454] T 59e507855e5043d38cef84e92adf437b P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 311f765f59754bdcb4eadc04e14bd4ea in term 1.
I20240502 01:54:14.399646 17435 leader_election.cc:302] T 59e507855e5043d38cef84e92adf437b P 311f765f59754bdcb4eadc04e14bd4ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b50a031f2784ba498b3e52e16916205, 311f765f59754bdcb4eadc04e14bd4ea; no voters: 
I20240502 01:54:14.399978 18863 raft_consensus.cc:2781] T 59e507855e5043d38cef84e92adf437b P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:14.400110 18863 raft_consensus.cc:686] T 59e507855e5043d38cef84e92adf437b P 311f765f59754bdcb4eadc04e14bd4ea [term 1 LEADER]: Becoming Leader. State: Replica: 311f765f59754bdcb4eadc04e14bd4ea, State: Running, Role: LEADER
I20240502 01:54:14.400398 18863 consensus_queue.cc:237] T 59e507855e5043d38cef84e92adf437b P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } }
I20240502 01:54:14.403520 17349 catalog_manager.cc:5484] T 59e507855e5043d38cef84e92adf437b P 311f765f59754bdcb4eadc04e14bd4ea reported cstate change: term changed from 0 to 1, leader changed from <none> to 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193). New cstate: current_term: 1 leader_uuid: "311f765f59754bdcb4eadc04e14bd4ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:14.433637 18810 raft_consensus.cc:482] T f07e9b433e554c85b3c25b7e708f8b2f P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:14.433841 18810 raft_consensus.cc:504] T f07e9b433e554c85b3c25b7e708f8b2f P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } }
I20240502 01:54:14.434509 18810 leader_election.cc:288] T f07e9b433e554c85b3c25b7e708f8b2f P 84d3a4b174c34e02aedab1e620731359 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193:35397), 1b50a031f2784ba498b3e52e16916205 (127.9.176.194:45617)
I20240502 01:54:14.435169 17500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f07e9b433e554c85b3c25b7e708f8b2f" candidate_uuid: "84d3a4b174c34e02aedab1e620731359" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "311f765f59754bdcb4eadc04e14bd4ea" is_pre_election: true
I20240502 01:54:14.435273 17634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f07e9b433e554c85b3c25b7e708f8b2f" candidate_uuid: "84d3a4b174c34e02aedab1e620731359" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b50a031f2784ba498b3e52e16916205" is_pre_election: true
I20240502 01:54:14.435421 17500 raft_consensus.cc:2454] T f07e9b433e554c85b3c25b7e708f8b2f P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84d3a4b174c34e02aedab1e620731359 in term 0.
I20240502 01:54:14.435498 17634 raft_consensus.cc:2454] T f07e9b433e554c85b3c25b7e708f8b2f P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84d3a4b174c34e02aedab1e620731359 in term 0.
I20240502 01:54:14.435974 17703 leader_election.cc:302] T f07e9b433e554c85b3c25b7e708f8b2f P 84d3a4b174c34e02aedab1e620731359 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b50a031f2784ba498b3e52e16916205, 84d3a4b174c34e02aedab1e620731359; no voters: 
I20240502 01:54:14.436308 18810 raft_consensus.cc:2781] T f07e9b433e554c85b3c25b7e708f8b2f P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:14.436410 18810 raft_consensus.cc:482] T f07e9b433e554c85b3c25b7e708f8b2f P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:14.436518 18810 raft_consensus.cc:3037] T f07e9b433e554c85b3c25b7e708f8b2f P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:14.438493 18810 raft_consensus.cc:504] T f07e9b433e554c85b3c25b7e708f8b2f P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } }
I20240502 01:54:14.439040 18810 leader_election.cc:288] T f07e9b433e554c85b3c25b7e708f8b2f P 84d3a4b174c34e02aedab1e620731359 [CANDIDATE]: Term 1 election: Requested vote from peers 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193:35397), 1b50a031f2784ba498b3e52e16916205 (127.9.176.194:45617)
I20240502 01:54:14.439519 17500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f07e9b433e554c85b3c25b7e708f8b2f" candidate_uuid: "84d3a4b174c34e02aedab1e620731359" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "311f765f59754bdcb4eadc04e14bd4ea"
I20240502 01:54:14.439656 17634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f07e9b433e554c85b3c25b7e708f8b2f" candidate_uuid: "84d3a4b174c34e02aedab1e620731359" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b50a031f2784ba498b3e52e16916205"
I20240502 01:54:14.439766 17500 raft_consensus.cc:3037] T f07e9b433e554c85b3c25b7e708f8b2f P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:14.439888 17634 raft_consensus.cc:3037] T f07e9b433e554c85b3c25b7e708f8b2f P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:14.441967 17500 raft_consensus.cc:2454] T f07e9b433e554c85b3c25b7e708f8b2f P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84d3a4b174c34e02aedab1e620731359 in term 1.
I20240502 01:54:14.442518 17703 leader_election.cc:302] T f07e9b433e554c85b3c25b7e708f8b2f P 84d3a4b174c34e02aedab1e620731359 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 311f765f59754bdcb4eadc04e14bd4ea, 84d3a4b174c34e02aedab1e620731359; no voters: 
I20240502 01:54:14.442562 17634 raft_consensus.cc:2454] T f07e9b433e554c85b3c25b7e708f8b2f P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84d3a4b174c34e02aedab1e620731359 in term 1.
I20240502 01:54:14.442905 18810 raft_consensus.cc:2781] T f07e9b433e554c85b3c25b7e708f8b2f P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:14.443084 18810 raft_consensus.cc:686] T f07e9b433e554c85b3c25b7e708f8b2f P 84d3a4b174c34e02aedab1e620731359 [term 1 LEADER]: Becoming Leader. State: Replica: 84d3a4b174c34e02aedab1e620731359, State: Running, Role: LEADER
I20240502 01:54:14.443369 18810 consensus_queue.cc:237] T f07e9b433e554c85b3c25b7e708f8b2f P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } }
I20240502 01:54:14.446568 17349 catalog_manager.cc:5484] T f07e9b433e554c85b3c25b7e708f8b2f P 84d3a4b174c34e02aedab1e620731359 reported cstate change: term changed from 0 to 1, leader changed from <none> to 84d3a4b174c34e02aedab1e620731359 (127.9.176.195). New cstate: current_term: 1 leader_uuid: "84d3a4b174c34e02aedab1e620731359" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:14.756386 18828 consensus_queue.cc:1035] T a5e03cf44eab41cc842342a138c04067 P da3584f429104d23a1a8899b35cc2138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:14.765524 17500 raft_consensus.cc:3037] T a5e03cf44eab41cc842342a138c04067 P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:14.777730 18828 consensus_queue.cc:1035] T a5e03cf44eab41cc842342a138c04067 P da3584f429104d23a1a8899b35cc2138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:54:14.878417 18886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:54:14.878897 18886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240502 01:54:14.878943 18863 consensus_queue.cc:1035] T 59e507855e5043d38cef84e92adf437b P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:14.884035 18863 consensus_queue.cc:1035] T 59e507855e5043d38cef84e92adf437b P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:54:14.892634 18886 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 01:54:14.921463 18810 consensus_queue.cc:1035] T f07e9b433e554c85b3c25b7e708f8b2f P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:14.927265 18810 consensus_queue.cc:1035] T f07e9b433e554c85b3c25b7e708f8b2f P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_4'

INSERT report
    rows total: 2000
    time total: 298.588 ms
  time per row: 0.149294 ms
-----------------------------------------------------
Suppressions used:
  count      bytes template
      2        596 libcrypto.so.1.1
-----------------------------------------------------

I20240502 01:54:16.302469 17347 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56698:
name: "rebalance_test_table_5"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 01:54:16.313757 17747 tablet_service.cc:1467] Processing CreateTablet for tablet b40f968655774ce887dff8b7f4651501 (DEFAULT_TABLE table=rebalance_test_table_5 [id=73ba62f6c85943bf85dfaaf459d924f0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:16.314294 17747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b40f968655774ce887dff8b7f4651501. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:16.314221 17612 tablet_service.cc:1467] Processing CreateTablet for tablet b40f968655774ce887dff8b7f4651501 (DEFAULT_TABLE table=rebalance_test_table_5 [id=73ba62f6c85943bf85dfaaf459d924f0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:16.314481 17878 tablet_service.cc:1467] Processing CreateTablet for tablet b40f968655774ce887dff8b7f4651501 (DEFAULT_TABLE table=rebalance_test_table_5 [id=73ba62f6c85943bf85dfaaf459d924f0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:16.314687 17612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b40f968655774ce887dff8b7f4651501. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:16.314929 17878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b40f968655774ce887dff8b7f4651501. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:16.316390 17478 tablet_service.cc:1467] Processing CreateTablet for tablet 38a60d7db47149ff90444e5276fc8935 (DEFAULT_TABLE table=rebalance_test_table_5 [id=73ba62f6c85943bf85dfaaf459d924f0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:16.316753 17478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38a60d7db47149ff90444e5276fc8935. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:16.320250 17880 tablet_service.cc:1467] Processing CreateTablet for tablet 17365abbb6c1430aa278719eb8e4a648 (DEFAULT_TABLE table=rebalance_test_table_5 [id=73ba62f6c85943bf85dfaaf459d924f0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:16.320822 17880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17365abbb6c1430aa278719eb8e4a648. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:16.325333 17746 tablet_service.cc:1467] Processing CreateTablet for tablet 17365abbb6c1430aa278719eb8e4a648 (DEFAULT_TABLE table=rebalance_test_table_5 [id=73ba62f6c85943bf85dfaaf459d924f0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:16.325878 17746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17365abbb6c1430aa278719eb8e4a648. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:16.328040 17745 tablet_service.cc:1467] Processing CreateTablet for tablet 38a60d7db47149ff90444e5276fc8935 (DEFAULT_TABLE table=rebalance_test_table_5 [id=73ba62f6c85943bf85dfaaf459d924f0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:16.328524 17745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38a60d7db47149ff90444e5276fc8935. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:16.330019 17611 tablet_service.cc:1467] Processing CreateTablet for tablet 17365abbb6c1430aa278719eb8e4a648 (DEFAULT_TABLE table=rebalance_test_table_5 [id=73ba62f6c85943bf85dfaaf459d924f0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:16.330591 17611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17365abbb6c1430aa278719eb8e4a648. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:16.332480 18960 tablet_bootstrap.cc:492] T 38a60d7db47149ff90444e5276fc8935 P 311f765f59754bdcb4eadc04e14bd4ea: Bootstrap starting.
I20240502 01:54:16.334249 17613 tablet_service.cc:1467] Processing CreateTablet for tablet 38a60d7db47149ff90444e5276fc8935 (DEFAULT_TABLE table=rebalance_test_table_5 [id=73ba62f6c85943bf85dfaaf459d924f0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:16.334731 17613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38a60d7db47149ff90444e5276fc8935. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:16.335106 18960 tablet_bootstrap.cc:654] T 38a60d7db47149ff90444e5276fc8935 P 311f765f59754bdcb4eadc04e14bd4ea: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:16.335283 18961 tablet_bootstrap.cc:492] T b40f968655774ce887dff8b7f4651501 P da3584f429104d23a1a8899b35cc2138: Bootstrap starting.
I20240502 01:54:16.337356 18961 tablet_bootstrap.cc:654] T b40f968655774ce887dff8b7f4651501 P da3584f429104d23a1a8899b35cc2138: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:16.340530 18960 tablet_bootstrap.cc:492] T 38a60d7db47149ff90444e5276fc8935 P 311f765f59754bdcb4eadc04e14bd4ea: No bootstrap required, opened a new log
I20240502 01:54:16.340620 18961 tablet_bootstrap.cc:492] T b40f968655774ce887dff8b7f4651501 P da3584f429104d23a1a8899b35cc2138: No bootstrap required, opened a new log
I20240502 01:54:16.340803 18961 ts_tablet_manager.cc:1367] T b40f968655774ce887dff8b7f4651501 P da3584f429104d23a1a8899b35cc2138: Time spent bootstrapping tablet: real 0.006s	user 0.004s	sys 0.000s
I20240502 01:54:16.340837 18960 ts_tablet_manager.cc:1367] T 38a60d7db47149ff90444e5276fc8935 P 311f765f59754bdcb4eadc04e14bd4ea: Time spent bootstrapping tablet: real 0.008s	user 0.008s	sys 0.000s
I20240502 01:54:16.341666 18961 raft_consensus.cc:348] T b40f968655774ce887dff8b7f4651501 P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } }
I20240502 01:54:16.341666 18960 raft_consensus.cc:348] T 38a60d7db47149ff90444e5276fc8935 P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } }
I20240502 01:54:16.341895 18960 raft_consensus.cc:374] T 38a60d7db47149ff90444e5276fc8935 P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:16.341946 18961 raft_consensus.cc:374] T b40f968655774ce887dff8b7f4651501 P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:16.342032 18960 raft_consensus.cc:729] T 38a60d7db47149ff90444e5276fc8935 P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 311f765f59754bdcb4eadc04e14bd4ea, State: Initialized, Role: FOLLOWER
I20240502 01:54:16.342082 18961 raft_consensus.cc:729] T b40f968655774ce887dff8b7f4651501 P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da3584f429104d23a1a8899b35cc2138, State: Initialized, Role: FOLLOWER
I20240502 01:54:16.342339 18960 consensus_queue.cc:260] T 38a60d7db47149ff90444e5276fc8935 P 311f765f59754bdcb4eadc04e14bd4ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } }
I20240502 01:54:16.342367 18961 consensus_queue.cc:260] T b40f968655774ce887dff8b7f4651501 P da3584f429104d23a1a8899b35cc2138 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } }
I20240502 01:54:16.342955 18961 ts_tablet_manager.cc:1398] T b40f968655774ce887dff8b7f4651501 P da3584f429104d23a1a8899b35cc2138: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240502 01:54:16.343060 18960 ts_tablet_manager.cc:1398] T 38a60d7db47149ff90444e5276fc8935 P 311f765f59754bdcb4eadc04e14bd4ea: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 01:54:16.343305 18961 tablet_bootstrap.cc:492] T 17365abbb6c1430aa278719eb8e4a648 P da3584f429104d23a1a8899b35cc2138: Bootstrap starting.
I20240502 01:54:16.346166 18959 tablet_bootstrap.cc:492] T b40f968655774ce887dff8b7f4651501 P 84d3a4b174c34e02aedab1e620731359: Bootstrap starting.
I20240502 01:54:16.347785 18959 tablet_bootstrap.cc:654] T b40f968655774ce887dff8b7f4651501 P 84d3a4b174c34e02aedab1e620731359: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:16.348901 18961 tablet_bootstrap.cc:654] T 17365abbb6c1430aa278719eb8e4a648 P da3584f429104d23a1a8899b35cc2138: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:16.351370 18961 tablet_bootstrap.cc:492] T 17365abbb6c1430aa278719eb8e4a648 P da3584f429104d23a1a8899b35cc2138: No bootstrap required, opened a new log
I20240502 01:54:16.351641 18961 ts_tablet_manager.cc:1367] T 17365abbb6c1430aa278719eb8e4a648 P da3584f429104d23a1a8899b35cc2138: Time spent bootstrapping tablet: real 0.008s	user 0.004s	sys 0.000s
I20240502 01:54:16.352644 18964 tablet_bootstrap.cc:492] T 17365abbb6c1430aa278719eb8e4a648 P 1b50a031f2784ba498b3e52e16916205: Bootstrap starting.
I20240502 01:54:16.352717 18961 raft_consensus.cc:348] T 17365abbb6c1430aa278719eb8e4a648 P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } }
I20240502 01:54:16.353796 18961 raft_consensus.cc:374] T 17365abbb6c1430aa278719eb8e4a648 P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:16.353966 18961 raft_consensus.cc:729] T 17365abbb6c1430aa278719eb8e4a648 P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da3584f429104d23a1a8899b35cc2138, State: Initialized, Role: FOLLOWER
I20240502 01:54:16.354424 18961 consensus_queue.cc:260] T 17365abbb6c1430aa278719eb8e4a648 P da3584f429104d23a1a8899b35cc2138 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } }
I20240502 01:54:16.354665 18964 tablet_bootstrap.cc:654] T 17365abbb6c1430aa278719eb8e4a648 P 1b50a031f2784ba498b3e52e16916205: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:16.355243 18961 ts_tablet_manager.cc:1398] T 17365abbb6c1430aa278719eb8e4a648 P da3584f429104d23a1a8899b35cc2138: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 01:54:16.357148 18959 tablet_bootstrap.cc:492] T b40f968655774ce887dff8b7f4651501 P 84d3a4b174c34e02aedab1e620731359: No bootstrap required, opened a new log
I20240502 01:54:16.357373 18959 ts_tablet_manager.cc:1367] T b40f968655774ce887dff8b7f4651501 P 84d3a4b174c34e02aedab1e620731359: Time spent bootstrapping tablet: real 0.011s	user 0.000s	sys 0.008s
I20240502 01:54:16.357463 18964 tablet_bootstrap.cc:492] T 17365abbb6c1430aa278719eb8e4a648 P 1b50a031f2784ba498b3e52e16916205: No bootstrap required, opened a new log
I20240502 01:54:16.357647 18964 ts_tablet_manager.cc:1367] T 17365abbb6c1430aa278719eb8e4a648 P 1b50a031f2784ba498b3e52e16916205: Time spent bootstrapping tablet: real 0.005s	user 0.004s	sys 0.000s
I20240502 01:54:16.358647 18959 raft_consensus.cc:348] T b40f968655774ce887dff8b7f4651501 P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } }
I20240502 01:54:16.358948 18959 raft_consensus.cc:374] T b40f968655774ce887dff8b7f4651501 P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:16.359068 18959 raft_consensus.cc:729] T b40f968655774ce887dff8b7f4651501 P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84d3a4b174c34e02aedab1e620731359, State: Initialized, Role: FOLLOWER
I20240502 01:54:16.358644 18964 raft_consensus.cc:348] T 17365abbb6c1430aa278719eb8e4a648 P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } }
I20240502 01:54:16.359221 18964 raft_consensus.cc:374] T 17365abbb6c1430aa278719eb8e4a648 P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:16.359400 18964 raft_consensus.cc:729] T 17365abbb6c1430aa278719eb8e4a648 P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b50a031f2784ba498b3e52e16916205, State: Initialized, Role: FOLLOWER
I20240502 01:54:16.359337 18959 consensus_queue.cc:260] T b40f968655774ce887dff8b7f4651501 P 84d3a4b174c34e02aedab1e620731359 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } }
I20240502 01:54:16.360090 18959 ts_tablet_manager.cc:1398] T b40f968655774ce887dff8b7f4651501 P 84d3a4b174c34e02aedab1e620731359: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240502 01:54:16.360397 18959 tablet_bootstrap.cc:492] T 17365abbb6c1430aa278719eb8e4a648 P 84d3a4b174c34e02aedab1e620731359: Bootstrap starting.
I20240502 01:54:16.360365 18964 consensus_queue.cc:260] T 17365abbb6c1430aa278719eb8e4a648 P 1b50a031f2784ba498b3e52e16916205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } }
I20240502 01:54:16.361575 18964 ts_tablet_manager.cc:1398] T 17365abbb6c1430aa278719eb8e4a648 P 1b50a031f2784ba498b3e52e16916205: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.000s
I20240502 01:54:16.361938 18964 tablet_bootstrap.cc:492] T b40f968655774ce887dff8b7f4651501 P 1b50a031f2784ba498b3e52e16916205: Bootstrap starting.
I20240502 01:54:16.362274 18959 tablet_bootstrap.cc:654] T 17365abbb6c1430aa278719eb8e4a648 P 84d3a4b174c34e02aedab1e620731359: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:16.363890 18964 tablet_bootstrap.cc:654] T b40f968655774ce887dff8b7f4651501 P 1b50a031f2784ba498b3e52e16916205: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:16.366282 18964 tablet_bootstrap.cc:492] T b40f968655774ce887dff8b7f4651501 P 1b50a031f2784ba498b3e52e16916205: No bootstrap required, opened a new log
I20240502 01:54:16.366533 18964 ts_tablet_manager.cc:1367] T b40f968655774ce887dff8b7f4651501 P 1b50a031f2784ba498b3e52e16916205: Time spent bootstrapping tablet: real 0.005s	user 0.004s	sys 0.000s
I20240502 01:54:16.367606 18964 raft_consensus.cc:348] T b40f968655774ce887dff8b7f4651501 P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } }
I20240502 01:54:16.367866 18964 raft_consensus.cc:374] T b40f968655774ce887dff8b7f4651501 P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:16.367970 18964 raft_consensus.cc:729] T b40f968655774ce887dff8b7f4651501 P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b50a031f2784ba498b3e52e16916205, State: Initialized, Role: FOLLOWER
I20240502 01:54:16.368373 18959 tablet_bootstrap.cc:492] T 17365abbb6c1430aa278719eb8e4a648 P 84d3a4b174c34e02aedab1e620731359: No bootstrap required, opened a new log
I20240502 01:54:16.368286 18964 consensus_queue.cc:260] T b40f968655774ce887dff8b7f4651501 P 1b50a031f2784ba498b3e52e16916205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } }
I20240502 01:54:16.368557 18959 ts_tablet_manager.cc:1367] T 17365abbb6c1430aa278719eb8e4a648 P 84d3a4b174c34e02aedab1e620731359: Time spent bootstrapping tablet: real 0.008s	user 0.003s	sys 0.003s
I20240502 01:54:16.368958 18964 ts_tablet_manager.cc:1398] T b40f968655774ce887dff8b7f4651501 P 1b50a031f2784ba498b3e52e16916205: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240502 01:54:16.369236 18959 raft_consensus.cc:348] T 17365abbb6c1430aa278719eb8e4a648 P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } }
I20240502 01:54:16.369320 18964 tablet_bootstrap.cc:492] T 38a60d7db47149ff90444e5276fc8935 P 1b50a031f2784ba498b3e52e16916205: Bootstrap starting.
I20240502 01:54:16.369415 18959 raft_consensus.cc:374] T 17365abbb6c1430aa278719eb8e4a648 P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:16.369500 18959 raft_consensus.cc:729] T 17365abbb6c1430aa278719eb8e4a648 P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84d3a4b174c34e02aedab1e620731359, State: Initialized, Role: FOLLOWER
I20240502 01:54:16.369791 18959 consensus_queue.cc:260] T 17365abbb6c1430aa278719eb8e4a648 P 84d3a4b174c34e02aedab1e620731359 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } }
I20240502 01:54:16.370229 18959 ts_tablet_manager.cc:1398] T 17365abbb6c1430aa278719eb8e4a648 P 84d3a4b174c34e02aedab1e620731359: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.001s
I20240502 01:54:16.370501 18959 tablet_bootstrap.cc:492] T 38a60d7db47149ff90444e5276fc8935 P 84d3a4b174c34e02aedab1e620731359: Bootstrap starting.
I20240502 01:54:16.371322 18964 tablet_bootstrap.cc:654] T 38a60d7db47149ff90444e5276fc8935 P 1b50a031f2784ba498b3e52e16916205: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:16.371798 18959 tablet_bootstrap.cc:654] T 38a60d7db47149ff90444e5276fc8935 P 84d3a4b174c34e02aedab1e620731359: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:16.378666 18964 tablet_bootstrap.cc:492] T 38a60d7db47149ff90444e5276fc8935 P 1b50a031f2784ba498b3e52e16916205: No bootstrap required, opened a new log
I20240502 01:54:16.378794 18959 tablet_bootstrap.cc:492] T 38a60d7db47149ff90444e5276fc8935 P 84d3a4b174c34e02aedab1e620731359: No bootstrap required, opened a new log
I20240502 01:54:16.378896 18964 ts_tablet_manager.cc:1367] T 38a60d7db47149ff90444e5276fc8935 P 1b50a031f2784ba498b3e52e16916205: Time spent bootstrapping tablet: real 0.010s	user 0.008s	sys 0.000s
I20240502 01:54:16.379016 18959 ts_tablet_manager.cc:1367] T 38a60d7db47149ff90444e5276fc8935 P 84d3a4b174c34e02aedab1e620731359: Time spent bootstrapping tablet: real 0.009s	user 0.004s	sys 0.004s
I20240502 01:54:16.379740 18959 raft_consensus.cc:348] T 38a60d7db47149ff90444e5276fc8935 P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } }
I20240502 01:54:16.379930 18959 raft_consensus.cc:374] T 38a60d7db47149ff90444e5276fc8935 P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:16.379818 18964 raft_consensus.cc:348] T 38a60d7db47149ff90444e5276fc8935 P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } }
I20240502 01:54:16.380038 18959 raft_consensus.cc:729] T 38a60d7db47149ff90444e5276fc8935 P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84d3a4b174c34e02aedab1e620731359, State: Initialized, Role: FOLLOWER
I20240502 01:54:16.380033 18964 raft_consensus.cc:374] T 38a60d7db47149ff90444e5276fc8935 P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:16.380129 18964 raft_consensus.cc:729] T 38a60d7db47149ff90444e5276fc8935 P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b50a031f2784ba498b3e52e16916205, State: Initialized, Role: FOLLOWER
I20240502 01:54:16.380278 18959 consensus_queue.cc:260] T 38a60d7db47149ff90444e5276fc8935 P 84d3a4b174c34e02aedab1e620731359 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } }
I20240502 01:54:16.380398 18964 consensus_queue.cc:260] T 38a60d7db47149ff90444e5276fc8935 P 1b50a031f2784ba498b3e52e16916205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } }
I20240502 01:54:16.380988 18959 ts_tablet_manager.cc:1398] T 38a60d7db47149ff90444e5276fc8935 P 84d3a4b174c34e02aedab1e620731359: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 01:54:16.381088 18964 ts_tablet_manager.cc:1398] T 38a60d7db47149ff90444e5276fc8935 P 1b50a031f2784ba498b3e52e16916205: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 01:54:16.432958 18828 raft_consensus.cc:482] T 17365abbb6c1430aa278719eb8e4a648 P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:16.433229 18828 raft_consensus.cc:504] T 17365abbb6c1430aa278719eb8e4a648 P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } }
I20240502 01:54:16.433900 18828 leader_election.cc:288] T 17365abbb6c1430aa278719eb8e4a648 P da3584f429104d23a1a8899b35cc2138 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84d3a4b174c34e02aedab1e620731359 (127.9.176.195:37357), 1b50a031f2784ba498b3e52e16916205 (127.9.176.194:45617)
I20240502 01:54:16.434525 17767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17365abbb6c1430aa278719eb8e4a648" candidate_uuid: "da3584f429104d23a1a8899b35cc2138" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84d3a4b174c34e02aedab1e620731359" is_pre_election: true
I20240502 01:54:16.434612 17633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17365abbb6c1430aa278719eb8e4a648" candidate_uuid: "da3584f429104d23a1a8899b35cc2138" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b50a031f2784ba498b3e52e16916205" is_pre_election: true
I20240502 01:54:16.434803 17767 raft_consensus.cc:2454] T 17365abbb6c1430aa278719eb8e4a648 P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da3584f429104d23a1a8899b35cc2138 in term 0.
I20240502 01:54:16.434852 17633 raft_consensus.cc:2454] T 17365abbb6c1430aa278719eb8e4a648 P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da3584f429104d23a1a8899b35cc2138 in term 0.
I20240502 01:54:16.435302 17835 leader_election.cc:302] T 17365abbb6c1430aa278719eb8e4a648 P da3584f429104d23a1a8899b35cc2138 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b50a031f2784ba498b3e52e16916205, da3584f429104d23a1a8899b35cc2138; no voters: 
I20240502 01:54:16.435631 18828 raft_consensus.cc:2781] T 17365abbb6c1430aa278719eb8e4a648 P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:16.435735 18828 raft_consensus.cc:482] T 17365abbb6c1430aa278719eb8e4a648 P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:16.435855 18828 raft_consensus.cc:3037] T 17365abbb6c1430aa278719eb8e4a648 P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:16.437728 18828 raft_consensus.cc:504] T 17365abbb6c1430aa278719eb8e4a648 P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } }
I20240502 01:54:16.438360 18828 leader_election.cc:288] T 17365abbb6c1430aa278719eb8e4a648 P da3584f429104d23a1a8899b35cc2138 [CANDIDATE]: Term 1 election: Requested vote from peers 84d3a4b174c34e02aedab1e620731359 (127.9.176.195:37357), 1b50a031f2784ba498b3e52e16916205 (127.9.176.194:45617)
I20240502 01:54:16.438874 17767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17365abbb6c1430aa278719eb8e4a648" candidate_uuid: "da3584f429104d23a1a8899b35cc2138" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84d3a4b174c34e02aedab1e620731359"
I20240502 01:54:16.438978 17633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17365abbb6c1430aa278719eb8e4a648" candidate_uuid: "da3584f429104d23a1a8899b35cc2138" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b50a031f2784ba498b3e52e16916205"
I20240502 01:54:16.439107 17767 raft_consensus.cc:3037] T 17365abbb6c1430aa278719eb8e4a648 P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:16.439222 17633 raft_consensus.cc:3037] T 17365abbb6c1430aa278719eb8e4a648 P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:16.442160 17633 raft_consensus.cc:2454] T 17365abbb6c1430aa278719eb8e4a648 P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da3584f429104d23a1a8899b35cc2138 in term 1.
I20240502 01:54:16.442184 17767 raft_consensus.cc:2454] T 17365abbb6c1430aa278719eb8e4a648 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da3584f429104d23a1a8899b35cc2138 in term 1.
I20240502 01:54:16.442663 17835 leader_election.cc:302] T 17365abbb6c1430aa278719eb8e4a648 P da3584f429104d23a1a8899b35cc2138 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b50a031f2784ba498b3e52e16916205, da3584f429104d23a1a8899b35cc2138; no voters: 
I20240502 01:54:16.442983 18828 raft_consensus.cc:2781] T 17365abbb6c1430aa278719eb8e4a648 P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:16.443436 18828 raft_consensus.cc:686] T 17365abbb6c1430aa278719eb8e4a648 P da3584f429104d23a1a8899b35cc2138 [term 1 LEADER]: Becoming Leader. State: Replica: da3584f429104d23a1a8899b35cc2138, State: Running, Role: LEADER
I20240502 01:54:16.443885 18828 consensus_queue.cc:237] T 17365abbb6c1430aa278719eb8e4a648 P da3584f429104d23a1a8899b35cc2138 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } }
I20240502 01:54:16.447530 17347 catalog_manager.cc:5484] T 17365abbb6c1430aa278719eb8e4a648 P da3584f429104d23a1a8899b35cc2138 reported cstate change: term changed from 0 to 1, leader changed from <none> to da3584f429104d23a1a8899b35cc2138 (127.9.176.196). New cstate: current_term: 1 leader_uuid: "da3584f429104d23a1a8899b35cc2138" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:16.470952 18615 raft_consensus.cc:482] T b40f968655774ce887dff8b7f4651501 P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:16.471225 18615 raft_consensus.cc:504] T b40f968655774ce887dff8b7f4651501 P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } }
I20240502 01:54:16.472110 18615 leader_election.cc:288] T b40f968655774ce887dff8b7f4651501 P 1b50a031f2784ba498b3e52e16916205 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84d3a4b174c34e02aedab1e620731359 (127.9.176.195:37357), da3584f429104d23a1a8899b35cc2138 (127.9.176.196:46381)
I20240502 01:54:16.472821 17900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b40f968655774ce887dff8b7f4651501" candidate_uuid: "1b50a031f2784ba498b3e52e16916205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da3584f429104d23a1a8899b35cc2138" is_pre_election: true
I20240502 01:54:16.473124 17900 raft_consensus.cc:2454] T b40f968655774ce887dff8b7f4651501 P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b50a031f2784ba498b3e52e16916205 in term 0.
I20240502 01:54:16.473664 17565 leader_election.cc:302] T b40f968655774ce887dff8b7f4651501 P 1b50a031f2784ba498b3e52e16916205 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b50a031f2784ba498b3e52e16916205, da3584f429104d23a1a8899b35cc2138; no voters: 
I20240502 01:54:16.474140 18615 raft_consensus.cc:2781] T b40f968655774ce887dff8b7f4651501 P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:16.474298 18615 raft_consensus.cc:482] T b40f968655774ce887dff8b7f4651501 P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:16.474432 18615 raft_consensus.cc:3037] T b40f968655774ce887dff8b7f4651501 P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:16.476764 18615 raft_consensus.cc:504] T b40f968655774ce887dff8b7f4651501 P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } }
I20240502 01:54:16.477723 18615 leader_election.cc:288] T b40f968655774ce887dff8b7f4651501 P 1b50a031f2784ba498b3e52e16916205 [CANDIDATE]: Term 1 election: Requested vote from peers 84d3a4b174c34e02aedab1e620731359 (127.9.176.195:37357), da3584f429104d23a1a8899b35cc2138 (127.9.176.196:46381)
I20240502 01:54:16.478335 17900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b40f968655774ce887dff8b7f4651501" candidate_uuid: "1b50a031f2784ba498b3e52e16916205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da3584f429104d23a1a8899b35cc2138"
I20240502 01:54:16.478561 17900 raft_consensus.cc:3037] T b40f968655774ce887dff8b7f4651501 P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:16.480177 17767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b40f968655774ce887dff8b7f4651501" candidate_uuid: "1b50a031f2784ba498b3e52e16916205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84d3a4b174c34e02aedab1e620731359"
I20240502 01:54:16.480486 17767 raft_consensus.cc:3037] T b40f968655774ce887dff8b7f4651501 P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:16.480974 17900 raft_consensus.cc:2454] T b40f968655774ce887dff8b7f4651501 P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b50a031f2784ba498b3e52e16916205 in term 1.
I20240502 01:54:16.480145 17766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b40f968655774ce887dff8b7f4651501" candidate_uuid: "1b50a031f2784ba498b3e52e16916205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84d3a4b174c34e02aedab1e620731359" is_pre_election: true
I20240502 01:54:16.481501 17565 leader_election.cc:302] T b40f968655774ce887dff8b7f4651501 P 1b50a031f2784ba498b3e52e16916205 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b50a031f2784ba498b3e52e16916205, da3584f429104d23a1a8899b35cc2138; no voters: 
I20240502 01:54:16.481892 18615 raft_consensus.cc:2781] T b40f968655774ce887dff8b7f4651501 P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:16.482396 18615 raft_consensus.cc:686] T b40f968655774ce887dff8b7f4651501 P 1b50a031f2784ba498b3e52e16916205 [term 1 LEADER]: Becoming Leader. State: Replica: 1b50a031f2784ba498b3e52e16916205, State: Running, Role: LEADER
I20240502 01:54:16.482743 18615 consensus_queue.cc:237] T b40f968655774ce887dff8b7f4651501 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } }
I20240502 01:54:16.483448 17767 raft_consensus.cc:2454] T b40f968655774ce887dff8b7f4651501 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b50a031f2784ba498b3e52e16916205 in term 1.
I20240502 01:54:16.486323 17347 catalog_manager.cc:5484] T b40f968655774ce887dff8b7f4651501 P 1b50a031f2784ba498b3e52e16916205 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b50a031f2784ba498b3e52e16916205 (127.9.176.194). New cstate: current_term: 1 leader_uuid: "1b50a031f2784ba498b3e52e16916205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:16.508339 18615 raft_consensus.cc:482] T 38a60d7db47149ff90444e5276fc8935 P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:16.508545 18615 raft_consensus.cc:504] T 38a60d7db47149ff90444e5276fc8935 P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } }
I20240502 01:54:16.509318 18615 leader_election.cc:288] T 38a60d7db47149ff90444e5276fc8935 P 1b50a031f2784ba498b3e52e16916205 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193:35397), 84d3a4b174c34e02aedab1e620731359 (127.9.176.195:37357)
I20240502 01:54:16.509977 17498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38a60d7db47149ff90444e5276fc8935" candidate_uuid: "1b50a031f2784ba498b3e52e16916205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "311f765f59754bdcb4eadc04e14bd4ea" is_pre_election: true
I20240502 01:54:16.509974 17767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38a60d7db47149ff90444e5276fc8935" candidate_uuid: "1b50a031f2784ba498b3e52e16916205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84d3a4b174c34e02aedab1e620731359" is_pre_election: true
I20240502 01:54:16.510207 17767 raft_consensus.cc:2454] T 38a60d7db47149ff90444e5276fc8935 P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b50a031f2784ba498b3e52e16916205 in term 0.
I20240502 01:54:16.510208 17498 raft_consensus.cc:2454] T 38a60d7db47149ff90444e5276fc8935 P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b50a031f2784ba498b3e52e16916205 in term 0.
I20240502 01:54:16.510646 17568 leader_election.cc:302] T 38a60d7db47149ff90444e5276fc8935 P 1b50a031f2784ba498b3e52e16916205 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b50a031f2784ba498b3e52e16916205, 84d3a4b174c34e02aedab1e620731359; no voters: 
I20240502 01:54:16.510937 18615 raft_consensus.cc:2781] T 38a60d7db47149ff90444e5276fc8935 P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:16.511077 18615 raft_consensus.cc:482] T 38a60d7db47149ff90444e5276fc8935 P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:16.511193 18615 raft_consensus.cc:3037] T 38a60d7db47149ff90444e5276fc8935 P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:16.537611 18615 raft_consensus.cc:504] T 38a60d7db47149ff90444e5276fc8935 P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } }
I20240502 01:54:16.538424 18615 leader_election.cc:288] T 38a60d7db47149ff90444e5276fc8935 P 1b50a031f2784ba498b3e52e16916205 [CANDIDATE]: Term 1 election: Requested vote from peers 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193:35397), 84d3a4b174c34e02aedab1e620731359 (127.9.176.195:37357)
I20240502 01:54:16.538936 17498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38a60d7db47149ff90444e5276fc8935" candidate_uuid: "1b50a031f2784ba498b3e52e16916205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "311f765f59754bdcb4eadc04e14bd4ea"
I20240502 01:54:16.539057 17767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38a60d7db47149ff90444e5276fc8935" candidate_uuid: "1b50a031f2784ba498b3e52e16916205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84d3a4b174c34e02aedab1e620731359"
I20240502 01:54:16.539184 17498 raft_consensus.cc:3037] T 38a60d7db47149ff90444e5276fc8935 P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:16.539316 17767 raft_consensus.cc:3037] T 38a60d7db47149ff90444e5276fc8935 P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:16.541374 17498 raft_consensus.cc:2454] T 38a60d7db47149ff90444e5276fc8935 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b50a031f2784ba498b3e52e16916205 in term 1.
I20240502 01:54:16.541467 17767 raft_consensus.cc:2454] T 38a60d7db47149ff90444e5276fc8935 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b50a031f2784ba498b3e52e16916205 in term 1.
I20240502 01:54:16.541975 17568 leader_election.cc:302] T 38a60d7db47149ff90444e5276fc8935 P 1b50a031f2784ba498b3e52e16916205 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b50a031f2784ba498b3e52e16916205, 311f765f59754bdcb4eadc04e14bd4ea; no voters: 
I20240502 01:54:16.542343 18615 raft_consensus.cc:2781] T 38a60d7db47149ff90444e5276fc8935 P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:16.542490 18615 raft_consensus.cc:686] T 38a60d7db47149ff90444e5276fc8935 P 1b50a031f2784ba498b3e52e16916205 [term 1 LEADER]: Becoming Leader. State: Replica: 1b50a031f2784ba498b3e52e16916205, State: Running, Role: LEADER
I20240502 01:54:16.542806 18615 consensus_queue.cc:237] T 38a60d7db47149ff90444e5276fc8935 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } }
I20240502 01:54:16.545848 17347 catalog_manager.cc:5484] T 38a60d7db47149ff90444e5276fc8935 P 1b50a031f2784ba498b3e52e16916205 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b50a031f2784ba498b3e52e16916205 (127.9.176.194). New cstate: current_term: 1 leader_uuid: "1b50a031f2784ba498b3e52e16916205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:16.899416 18828 consensus_queue.cc:1035] T 17365abbb6c1430aa278719eb8e4a648 P da3584f429104d23a1a8899b35cc2138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:16.907593 18615 consensus_queue.cc:1035] T b40f968655774ce887dff8b7f4651501 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:16.909377 18973 consensus_queue.cc:1035] T 17365abbb6c1430aa278719eb8e4a648 P da3584f429104d23a1a8899b35cc2138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:16.939004 18978 consensus_queue.cc:1035] T b40f968655774ce887dff8b7f4651501 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:54:16.986148 18982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:54:16.986559 18982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:54:16.994566 18982 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'
I20240502 01:54:17.107204 18615 consensus_queue.cc:1035] T 38a60d7db47149ff90444e5276fc8935 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:17.109464 18615 consensus_queue.cc:1035] T 38a60d7db47149ff90444e5276fc8935 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:54:17.293805 17565 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.176.196:46381, user_credentials={real_user=slave}} blocked reactor thread for 63059.1us

INSERT report
    rows total: 2000
    time total: 269.524 ms
  time per row: 0.134762 ms
-----------------------------------------------------
Suppressions used:
  count      bytes template
      4       1192 libcrypto.so.1.1
-----------------------------------------------------

I20240502 01:54:18.382337 17353 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56698:
name: "rebalance_test_table_6"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 01:54:18.392509 17611 tablet_service.cc:1467] Processing CreateTablet for tablet 075ded9523e44c3d92844805338643a3 (DEFAULT_TABLE table=rebalance_test_table_6 [id=d49bf19a92cf45ceadf02d01dbdae6ad]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:18.392850 17478 tablet_service.cc:1467] Processing CreateTablet for tablet 075ded9523e44c3d92844805338643a3 (DEFAULT_TABLE table=rebalance_test_table_6 [id=d49bf19a92cf45ceadf02d01dbdae6ad]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:18.393047 17611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 075ded9523e44c3d92844805338643a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:18.393318 17478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 075ded9523e44c3d92844805338643a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:18.394017 17747 tablet_service.cc:1467] Processing CreateTablet for tablet 60ff2e50b1d74e0c8e0890b87f9ac8ba (DEFAULT_TABLE table=rebalance_test_table_6 [id=d49bf19a92cf45ceadf02d01dbdae6ad]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:18.394150 17612 tablet_service.cc:1467] Processing CreateTablet for tablet a6b5231a18a5414dbd010d4d48f08dbd (DEFAULT_TABLE table=rebalance_test_table_6 [id=d49bf19a92cf45ceadf02d01dbdae6ad]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:18.394479 17747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60ff2e50b1d74e0c8e0890b87f9ac8ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:18.394556 17612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6b5231a18a5414dbd010d4d48f08dbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:18.400585 17878 tablet_service.cc:1467] Processing CreateTablet for tablet 075ded9523e44c3d92844805338643a3 (DEFAULT_TABLE table=rebalance_test_table_6 [id=d49bf19a92cf45ceadf02d01dbdae6ad]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:18.400640 17880 tablet_service.cc:1467] Processing CreateTablet for tablet 60ff2e50b1d74e0c8e0890b87f9ac8ba (DEFAULT_TABLE table=rebalance_test_table_6 [id=d49bf19a92cf45ceadf02d01dbdae6ad]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:18.401075 17878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 075ded9523e44c3d92844805338643a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:18.401345 17880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60ff2e50b1d74e0c8e0890b87f9ac8ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:18.403245 19048 tablet_bootstrap.cc:492] T 075ded9523e44c3d92844805338643a3 P 311f765f59754bdcb4eadc04e14bd4ea: Bootstrap starting.
I20240502 01:54:18.403271 19049 tablet_bootstrap.cc:492] T 075ded9523e44c3d92844805338643a3 P 1b50a031f2784ba498b3e52e16916205: Bootstrap starting.
I20240502 01:54:18.405066 19049 tablet_bootstrap.cc:654] T 075ded9523e44c3d92844805338643a3 P 1b50a031f2784ba498b3e52e16916205: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:18.405083 19048 tablet_bootstrap.cc:654] T 075ded9523e44c3d92844805338643a3 P 311f765f59754bdcb4eadc04e14bd4ea: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:18.408183 17879 tablet_service.cc:1467] Processing CreateTablet for tablet a6b5231a18a5414dbd010d4d48f08dbd (DEFAULT_TABLE table=rebalance_test_table_6 [id=d49bf19a92cf45ceadf02d01dbdae6ad]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:18.408835 17879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6b5231a18a5414dbd010d4d48f08dbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:18.411017 17477 tablet_service.cc:1467] Processing CreateTablet for tablet 60ff2e50b1d74e0c8e0890b87f9ac8ba (DEFAULT_TABLE table=rebalance_test_table_6 [id=d49bf19a92cf45ceadf02d01dbdae6ad]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:18.411499 17477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60ff2e50b1d74e0c8e0890b87f9ac8ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:18.416163 17479 tablet_service.cc:1467] Processing CreateTablet for tablet a6b5231a18a5414dbd010d4d48f08dbd (DEFAULT_TABLE table=rebalance_test_table_6 [id=d49bf19a92cf45ceadf02d01dbdae6ad]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:18.416733 17479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6b5231a18a5414dbd010d4d48f08dbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:18.424109 19053 tablet_bootstrap.cc:492] T 075ded9523e44c3d92844805338643a3 P da3584f429104d23a1a8899b35cc2138: Bootstrap starting.
I20240502 01:54:18.427558 19053 tablet_bootstrap.cc:654] T 075ded9523e44c3d92844805338643a3 P da3584f429104d23a1a8899b35cc2138: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:18.429495 19052 tablet_bootstrap.cc:492] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 84d3a4b174c34e02aedab1e620731359: Bootstrap starting.
I20240502 01:54:18.430531 19049 tablet_bootstrap.cc:492] T 075ded9523e44c3d92844805338643a3 P 1b50a031f2784ba498b3e52e16916205: No bootstrap required, opened a new log
I20240502 01:54:18.430863 19049 ts_tablet_manager.cc:1367] T 075ded9523e44c3d92844805338643a3 P 1b50a031f2784ba498b3e52e16916205: Time spent bootstrapping tablet: real 0.028s	user 0.004s	sys 0.004s
I20240502 01:54:18.431418 19052 tablet_bootstrap.cc:654] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 84d3a4b174c34e02aedab1e620731359: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:18.431504 19048 tablet_bootstrap.cc:492] T 075ded9523e44c3d92844805338643a3 P 311f765f59754bdcb4eadc04e14bd4ea: No bootstrap required, opened a new log
I20240502 01:54:18.431736 19048 ts_tablet_manager.cc:1367] T 075ded9523e44c3d92844805338643a3 P 311f765f59754bdcb4eadc04e14bd4ea: Time spent bootstrapping tablet: real 0.029s	user 0.023s	sys 0.004s
I20240502 01:54:18.431947 19049 raft_consensus.cc:348] T 075ded9523e44c3d92844805338643a3 P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } }
I20240502 01:54:18.432216 19049 raft_consensus.cc:374] T 075ded9523e44c3d92844805338643a3 P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:18.432325 19049 raft_consensus.cc:729] T 075ded9523e44c3d92844805338643a3 P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b50a031f2784ba498b3e52e16916205, State: Initialized, Role: FOLLOWER
I20240502 01:54:18.432631 19048 raft_consensus.cc:348] T 075ded9523e44c3d92844805338643a3 P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } }
I20240502 01:54:18.432580 19049 consensus_queue.cc:260] T 075ded9523e44c3d92844805338643a3 P 1b50a031f2784ba498b3e52e16916205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } }
I20240502 01:54:18.432897 19048 raft_consensus.cc:374] T 075ded9523e44c3d92844805338643a3 P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:18.433001 19048 raft_consensus.cc:729] T 075ded9523e44c3d92844805338643a3 P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 311f765f59754bdcb4eadc04e14bd4ea, State: Initialized, Role: FOLLOWER
I20240502 01:54:18.433195 19053 tablet_bootstrap.cc:492] T 075ded9523e44c3d92844805338643a3 P da3584f429104d23a1a8899b35cc2138: No bootstrap required, opened a new log
I20240502 01:54:18.433398 19053 ts_tablet_manager.cc:1367] T 075ded9523e44c3d92844805338643a3 P da3584f429104d23a1a8899b35cc2138: Time spent bootstrapping tablet: real 0.009s	user 0.002s	sys 0.002s
I20240502 01:54:18.433326 19048 consensus_queue.cc:260] T 075ded9523e44c3d92844805338643a3 P 311f765f59754bdcb4eadc04e14bd4ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } }
I20240502 01:54:18.433701 19049 ts_tablet_manager.cc:1398] T 075ded9523e44c3d92844805338643a3 P 1b50a031f2784ba498b3e52e16916205: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 01:54:18.433995 19049 tablet_bootstrap.cc:492] T a6b5231a18a5414dbd010d4d48f08dbd P 1b50a031f2784ba498b3e52e16916205: Bootstrap starting.
I20240502 01:54:18.434767 19048 ts_tablet_manager.cc:1398] T 075ded9523e44c3d92844805338643a3 P 311f765f59754bdcb4eadc04e14bd4ea: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 01:54:18.435083 19048 tablet_bootstrap.cc:492] T a6b5231a18a5414dbd010d4d48f08dbd P 311f765f59754bdcb4eadc04e14bd4ea: Bootstrap starting.
I20240502 01:54:18.435528 19053 raft_consensus.cc:348] T 075ded9523e44c3d92844805338643a3 P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } }
I20240502 01:54:18.435851 19053 raft_consensus.cc:374] T 075ded9523e44c3d92844805338643a3 P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:18.435993 19053 raft_consensus.cc:729] T 075ded9523e44c3d92844805338643a3 P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da3584f429104d23a1a8899b35cc2138, State: Initialized, Role: FOLLOWER
I20240502 01:54:18.436302 19053 consensus_queue.cc:260] T 075ded9523e44c3d92844805338643a3 P da3584f429104d23a1a8899b35cc2138 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } }
I20240502 01:54:18.438082 19048 tablet_bootstrap.cc:654] T a6b5231a18a5414dbd010d4d48f08dbd P 311f765f59754bdcb4eadc04e14bd4ea: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:18.438858 19053 ts_tablet_manager.cc:1398] T 075ded9523e44c3d92844805338643a3 P da3584f429104d23a1a8899b35cc2138: Time spent starting tablet: real 0.005s	user 0.001s	sys 0.001s
I20240502 01:54:18.439261 19053 tablet_bootstrap.cc:492] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P da3584f429104d23a1a8899b35cc2138: Bootstrap starting.
I20240502 01:54:18.444332 19053 tablet_bootstrap.cc:654] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P da3584f429104d23a1a8899b35cc2138: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:18.444451 19049 tablet_bootstrap.cc:654] T a6b5231a18a5414dbd010d4d48f08dbd P 1b50a031f2784ba498b3e52e16916205: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:18.446656 19052 tablet_bootstrap.cc:492] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 84d3a4b174c34e02aedab1e620731359: No bootstrap required, opened a new log
I20240502 01:54:18.446869 19052 ts_tablet_manager.cc:1367] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 84d3a4b174c34e02aedab1e620731359: Time spent bootstrapping tablet: real 0.018s	user 0.010s	sys 0.000s
I20240502 01:54:18.448349 19048 tablet_bootstrap.cc:492] T a6b5231a18a5414dbd010d4d48f08dbd P 311f765f59754bdcb4eadc04e14bd4ea: No bootstrap required, opened a new log
I20240502 01:54:18.448207 19052 raft_consensus.cc:348] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } }
I20240502 01:54:18.448496 19052 raft_consensus.cc:374] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:18.448671 19052 raft_consensus.cc:729] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84d3a4b174c34e02aedab1e620731359, State: Initialized, Role: FOLLOWER
I20240502 01:54:18.448694 19048 ts_tablet_manager.cc:1367] T a6b5231a18a5414dbd010d4d48f08dbd P 311f765f59754bdcb4eadc04e14bd4ea: Time spent bootstrapping tablet: real 0.014s	user 0.007s	sys 0.003s
I20240502 01:54:18.449031 19052 consensus_queue.cc:260] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 84d3a4b174c34e02aedab1e620731359 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } }
I20240502 01:54:18.449781 19048 raft_consensus.cc:348] T a6b5231a18a5414dbd010d4d48f08dbd P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } }
I20240502 01:54:18.450093 19048 raft_consensus.cc:374] T a6b5231a18a5414dbd010d4d48f08dbd P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:18.450179 19048 raft_consensus.cc:729] T a6b5231a18a5414dbd010d4d48f08dbd P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 311f765f59754bdcb4eadc04e14bd4ea, State: Initialized, Role: FOLLOWER
I20240502 01:54:18.450484 19048 consensus_queue.cc:260] T a6b5231a18a5414dbd010d4d48f08dbd P 311f765f59754bdcb4eadc04e14bd4ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } }
I20240502 01:54:18.453486 19052 ts_tablet_manager.cc:1398] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 84d3a4b174c34e02aedab1e620731359: Time spent starting tablet: real 0.006s	user 0.000s	sys 0.003s
I20240502 01:54:18.454807 19048 ts_tablet_manager.cc:1398] T a6b5231a18a5414dbd010d4d48f08dbd P 311f765f59754bdcb4eadc04e14bd4ea: Time spent starting tablet: real 0.006s	user 0.002s	sys 0.000s
I20240502 01:54:18.457468 19048 tablet_bootstrap.cc:492] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 311f765f59754bdcb4eadc04e14bd4ea: Bootstrap starting.
I20240502 01:54:18.458094 19049 tablet_bootstrap.cc:492] T a6b5231a18a5414dbd010d4d48f08dbd P 1b50a031f2784ba498b3e52e16916205: No bootstrap required, opened a new log
I20240502 01:54:18.458098 19053 tablet_bootstrap.cc:492] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P da3584f429104d23a1a8899b35cc2138: No bootstrap required, opened a new log
I20240502 01:54:18.458287 19053 ts_tablet_manager.cc:1367] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P da3584f429104d23a1a8899b35cc2138: Time spent bootstrapping tablet: real 0.019s	user 0.008s	sys 0.005s
I20240502 01:54:18.458299 19049 ts_tablet_manager.cc:1367] T a6b5231a18a5414dbd010d4d48f08dbd P 1b50a031f2784ba498b3e52e16916205: Time spent bootstrapping tablet: real 0.024s	user 0.005s	sys 0.008s
I20240502 01:54:18.459146 19049 raft_consensus.cc:348] T a6b5231a18a5414dbd010d4d48f08dbd P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } }
I20240502 01:54:18.459373 19048 tablet_bootstrap.cc:654] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 311f765f59754bdcb4eadc04e14bd4ea: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:18.459426 19049 raft_consensus.cc:374] T a6b5231a18a5414dbd010d4d48f08dbd P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:18.459326 19053 raft_consensus.cc:348] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } }
I20240502 01:54:18.459496 19049 raft_consensus.cc:729] T a6b5231a18a5414dbd010d4d48f08dbd P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b50a031f2784ba498b3e52e16916205, State: Initialized, Role: FOLLOWER
I20240502 01:54:18.459553 19053 raft_consensus.cc:374] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:18.459662 19053 raft_consensus.cc:729] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da3584f429104d23a1a8899b35cc2138, State: Initialized, Role: FOLLOWER
I20240502 01:54:18.459828 19049 consensus_queue.cc:260] T a6b5231a18a5414dbd010d4d48f08dbd P 1b50a031f2784ba498b3e52e16916205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } }
I20240502 01:54:18.459966 19053 consensus_queue.cc:260] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P da3584f429104d23a1a8899b35cc2138 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } }
I20240502 01:54:18.460582 19053 ts_tablet_manager.cc:1398] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P da3584f429104d23a1a8899b35cc2138: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.000s
I20240502 01:54:18.461879 19049 ts_tablet_manager.cc:1398] T a6b5231a18a5414dbd010d4d48f08dbd P 1b50a031f2784ba498b3e52e16916205: Time spent starting tablet: real 0.003s	user 0.003s	sys 0.000s
I20240502 01:54:18.461009 19053 tablet_bootstrap.cc:492] T a6b5231a18a5414dbd010d4d48f08dbd P da3584f429104d23a1a8899b35cc2138: Bootstrap starting.
I20240502 01:54:18.462450 19048 tablet_bootstrap.cc:492] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 311f765f59754bdcb4eadc04e14bd4ea: No bootstrap required, opened a new log
I20240502 01:54:18.462636 19048 ts_tablet_manager.cc:1367] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 311f765f59754bdcb4eadc04e14bd4ea: Time spent bootstrapping tablet: real 0.005s	user 0.001s	sys 0.004s
I20240502 01:54:18.464152 19053 tablet_bootstrap.cc:654] T a6b5231a18a5414dbd010d4d48f08dbd P da3584f429104d23a1a8899b35cc2138: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:18.464380 19048 raft_consensus.cc:348] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } }
I20240502 01:54:18.464648 19048 raft_consensus.cc:374] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:18.464766 19048 raft_consensus.cc:729] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 311f765f59754bdcb4eadc04e14bd4ea, State: Initialized, Role: FOLLOWER
I20240502 01:54:18.465052 19048 consensus_queue.cc:260] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 311f765f59754bdcb4eadc04e14bd4ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } }
I20240502 01:54:18.466241 19053 tablet_bootstrap.cc:492] T a6b5231a18a5414dbd010d4d48f08dbd P da3584f429104d23a1a8899b35cc2138: No bootstrap required, opened a new log
I20240502 01:54:18.466403 19053 ts_tablet_manager.cc:1367] T a6b5231a18a5414dbd010d4d48f08dbd P da3584f429104d23a1a8899b35cc2138: Time spent bootstrapping tablet: real 0.005s	user 0.003s	sys 0.000s
I20240502 01:54:18.466514 19048 ts_tablet_manager.cc:1398] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 311f765f59754bdcb4eadc04e14bd4ea: Time spent starting tablet: real 0.004s	user 0.001s	sys 0.001s
I20240502 01:54:18.467073 19053 raft_consensus.cc:348] T a6b5231a18a5414dbd010d4d48f08dbd P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } }
I20240502 01:54:18.467237 19053 raft_consensus.cc:374] T a6b5231a18a5414dbd010d4d48f08dbd P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:18.467339 19053 raft_consensus.cc:729] T a6b5231a18a5414dbd010d4d48f08dbd P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da3584f429104d23a1a8899b35cc2138, State: Initialized, Role: FOLLOWER
I20240502 01:54:18.467638 19053 consensus_queue.cc:260] T a6b5231a18a5414dbd010d4d48f08dbd P da3584f429104d23a1a8899b35cc2138 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } }
I20240502 01:54:18.468310 19053 ts_tablet_manager.cc:1398] T a6b5231a18a5414dbd010d4d48f08dbd P da3584f429104d23a1a8899b35cc2138: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 01:54:18.541436 18972 raft_consensus.cc:482] T a6b5231a18a5414dbd010d4d48f08dbd P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:18.541661 18972 raft_consensus.cc:504] T a6b5231a18a5414dbd010d4d48f08dbd P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } }
I20240502 01:54:18.542364 18972 leader_election.cc:288] T a6b5231a18a5414dbd010d4d48f08dbd P 311f765f59754bdcb4eadc04e14bd4ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b50a031f2784ba498b3e52e16916205 (127.9.176.194:45617), da3584f429104d23a1a8899b35cc2138 (127.9.176.196:46381)
I20240502 01:54:18.542941 17900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6b5231a18a5414dbd010d4d48f08dbd" candidate_uuid: "311f765f59754bdcb4eadc04e14bd4ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da3584f429104d23a1a8899b35cc2138" is_pre_election: true
I20240502 01:54:18.542982 17634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6b5231a18a5414dbd010d4d48f08dbd" candidate_uuid: "311f765f59754bdcb4eadc04e14bd4ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b50a031f2784ba498b3e52e16916205" is_pre_election: true
I20240502 01:54:18.543231 17900 raft_consensus.cc:2454] T a6b5231a18a5414dbd010d4d48f08dbd P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 311f765f59754bdcb4eadc04e14bd4ea in term 0.
I20240502 01:54:18.543231 17634 raft_consensus.cc:2454] T a6b5231a18a5414dbd010d4d48f08dbd P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 311f765f59754bdcb4eadc04e14bd4ea in term 0.
I20240502 01:54:18.543805 17432 leader_election.cc:302] T a6b5231a18a5414dbd010d4d48f08dbd P 311f765f59754bdcb4eadc04e14bd4ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 311f765f59754bdcb4eadc04e14bd4ea, da3584f429104d23a1a8899b35cc2138; no voters: 
I20240502 01:54:18.544165 18972 raft_consensus.cc:2781] T a6b5231a18a5414dbd010d4d48f08dbd P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:18.544447 18972 raft_consensus.cc:482] T a6b5231a18a5414dbd010d4d48f08dbd P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:18.544581 18972 raft_consensus.cc:3037] T a6b5231a18a5414dbd010d4d48f08dbd P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:18.546938 18972 raft_consensus.cc:504] T a6b5231a18a5414dbd010d4d48f08dbd P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } }
I20240502 01:54:18.547760 18972 leader_election.cc:288] T a6b5231a18a5414dbd010d4d48f08dbd P 311f765f59754bdcb4eadc04e14bd4ea [CANDIDATE]: Term 1 election: Requested vote from peers 1b50a031f2784ba498b3e52e16916205 (127.9.176.194:45617), da3584f429104d23a1a8899b35cc2138 (127.9.176.196:46381)
I20240502 01:54:18.548218 17634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6b5231a18a5414dbd010d4d48f08dbd" candidate_uuid: "311f765f59754bdcb4eadc04e14bd4ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b50a031f2784ba498b3e52e16916205"
I20240502 01:54:18.548239 17900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6b5231a18a5414dbd010d4d48f08dbd" candidate_uuid: "311f765f59754bdcb4eadc04e14bd4ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da3584f429104d23a1a8899b35cc2138"
I20240502 01:54:18.548447 17900 raft_consensus.cc:3037] T a6b5231a18a5414dbd010d4d48f08dbd P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:18.548446 17634 raft_consensus.cc:3037] T a6b5231a18a5414dbd010d4d48f08dbd P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:18.550482 17634 raft_consensus.cc:2454] T a6b5231a18a5414dbd010d4d48f08dbd P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 311f765f59754bdcb4eadc04e14bd4ea in term 1.
I20240502 01:54:18.550483 17900 raft_consensus.cc:2454] T a6b5231a18a5414dbd010d4d48f08dbd P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 311f765f59754bdcb4eadc04e14bd4ea in term 1.
I20240502 01:54:18.550989 17432 leader_election.cc:302] T a6b5231a18a5414dbd010d4d48f08dbd P 311f765f59754bdcb4eadc04e14bd4ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1b50a031f2784ba498b3e52e16916205, 311f765f59754bdcb4eadc04e14bd4ea, da3584f429104d23a1a8899b35cc2138; no voters: 
I20240502 01:54:18.551323 18972 raft_consensus.cc:2781] T a6b5231a18a5414dbd010d4d48f08dbd P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:18.551460 18972 raft_consensus.cc:686] T a6b5231a18a5414dbd010d4d48f08dbd P 311f765f59754bdcb4eadc04e14bd4ea [term 1 LEADER]: Becoming Leader. State: Replica: 311f765f59754bdcb4eadc04e14bd4ea, State: Running, Role: LEADER
I20240502 01:54:18.551751 18972 consensus_queue.cc:237] T a6b5231a18a5414dbd010d4d48f08dbd P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } }
I20240502 01:54:18.555050 17347 catalog_manager.cc:5484] T a6b5231a18a5414dbd010d4d48f08dbd P 311f765f59754bdcb4eadc04e14bd4ea reported cstate change: term changed from 0 to 1, leader changed from <none> to 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193). New cstate: current_term: 1 leader_uuid: "311f765f59754bdcb4eadc04e14bd4ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:18.597879 18972 raft_consensus.cc:482] T 075ded9523e44c3d92844805338643a3 P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:18.598109 18972 raft_consensus.cc:504] T 075ded9523e44c3d92844805338643a3 P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } }
I20240502 01:54:18.598891 18972 leader_election.cc:288] T 075ded9523e44c3d92844805338643a3 P 311f765f59754bdcb4eadc04e14bd4ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b50a031f2784ba498b3e52e16916205 (127.9.176.194:45617), da3584f429104d23a1a8899b35cc2138 (127.9.176.196:46381)
I20240502 01:54:18.599493 17900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "075ded9523e44c3d92844805338643a3" candidate_uuid: "311f765f59754bdcb4eadc04e14bd4ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da3584f429104d23a1a8899b35cc2138" is_pre_election: true
I20240502 01:54:18.599495 17634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "075ded9523e44c3d92844805338643a3" candidate_uuid: "311f765f59754bdcb4eadc04e14bd4ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b50a031f2784ba498b3e52e16916205" is_pre_election: true
I20240502 01:54:18.599774 17900 raft_consensus.cc:2454] T 075ded9523e44c3d92844805338643a3 P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 311f765f59754bdcb4eadc04e14bd4ea in term 0.
I20240502 01:54:18.599778 17634 raft_consensus.cc:2454] T 075ded9523e44c3d92844805338643a3 P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 311f765f59754bdcb4eadc04e14bd4ea in term 0.
I20240502 01:54:18.600210 17432 leader_election.cc:302] T 075ded9523e44c3d92844805338643a3 P 311f765f59754bdcb4eadc04e14bd4ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 311f765f59754bdcb4eadc04e14bd4ea, da3584f429104d23a1a8899b35cc2138; no voters: 
I20240502 01:54:18.600507 18972 raft_consensus.cc:2781] T 075ded9523e44c3d92844805338643a3 P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:18.600620 18972 raft_consensus.cc:482] T 075ded9523e44c3d92844805338643a3 P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:18.600740 18972 raft_consensus.cc:3037] T 075ded9523e44c3d92844805338643a3 P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:18.602947 18972 raft_consensus.cc:504] T 075ded9523e44c3d92844805338643a3 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } }
I20240502 01:54:18.603719 18972 leader_election.cc:288] T 075ded9523e44c3d92844805338643a3 P 311f765f59754bdcb4eadc04e14bd4ea [CANDIDATE]: Term 1 election: Requested vote from peers 1b50a031f2784ba498b3e52e16916205 (127.9.176.194:45617), da3584f429104d23a1a8899b35cc2138 (127.9.176.196:46381)
I20240502 01:54:18.604173 17633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "075ded9523e44c3d92844805338643a3" candidate_uuid: "311f765f59754bdcb4eadc04e14bd4ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b50a031f2784ba498b3e52e16916205"
I20240502 01:54:18.604398 17633 raft_consensus.cc:3037] T 075ded9523e44c3d92844805338643a3 P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:18.604220 17900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "075ded9523e44c3d92844805338643a3" candidate_uuid: "311f765f59754bdcb4eadc04e14bd4ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da3584f429104d23a1a8899b35cc2138"
I20240502 01:54:18.604596 17900 raft_consensus.cc:3037] T 075ded9523e44c3d92844805338643a3 P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:18.606456 17633 raft_consensus.cc:2454] T 075ded9523e44c3d92844805338643a3 P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 311f765f59754bdcb4eadc04e14bd4ea in term 1.
I20240502 01:54:18.606557 17900 raft_consensus.cc:2454] T 075ded9523e44c3d92844805338643a3 P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 311f765f59754bdcb4eadc04e14bd4ea in term 1.
I20240502 01:54:18.607043 17432 leader_election.cc:302] T 075ded9523e44c3d92844805338643a3 P 311f765f59754bdcb4eadc04e14bd4ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 311f765f59754bdcb4eadc04e14bd4ea, da3584f429104d23a1a8899b35cc2138; no voters: 
I20240502 01:54:18.607390 18972 raft_consensus.cc:2781] T 075ded9523e44c3d92844805338643a3 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:18.607606 18972 raft_consensus.cc:686] T 075ded9523e44c3d92844805338643a3 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 LEADER]: Becoming Leader. State: Replica: 311f765f59754bdcb4eadc04e14bd4ea, State: Running, Role: LEADER
I20240502 01:54:18.607892 18972 consensus_queue.cc:237] T 075ded9523e44c3d92844805338643a3 P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } }
I20240502 01:54:18.611299 17347 catalog_manager.cc:5484] T 075ded9523e44c3d92844805338643a3 P 311f765f59754bdcb4eadc04e14bd4ea reported cstate change: term changed from 0 to 1, leader changed from <none> to 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193). New cstate: current_term: 1 leader_uuid: "311f765f59754bdcb4eadc04e14bd4ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:18.622447 18828 raft_consensus.cc:482] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:18.622705 18828 raft_consensus.cc:504] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } }
I20240502 01:54:18.623374 18828 leader_election.cc:288] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P da3584f429104d23a1a8899b35cc2138 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193:35397), 84d3a4b174c34e02aedab1e620731359 (127.9.176.195:37357)
I20240502 01:54:18.623988 17498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60ff2e50b1d74e0c8e0890b87f9ac8ba" candidate_uuid: "da3584f429104d23a1a8899b35cc2138" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "311f765f59754bdcb4eadc04e14bd4ea" is_pre_election: true
I20240502 01:54:18.624037 17766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60ff2e50b1d74e0c8e0890b87f9ac8ba" candidate_uuid: "da3584f429104d23a1a8899b35cc2138" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84d3a4b174c34e02aedab1e620731359" is_pre_election: true
I20240502 01:54:18.624248 17498 raft_consensus.cc:2454] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da3584f429104d23a1a8899b35cc2138 in term 0.
I20240502 01:54:18.624258 17766 raft_consensus.cc:2454] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da3584f429104d23a1a8899b35cc2138 in term 0.
I20240502 01:54:18.624729 17835 leader_election.cc:302] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P da3584f429104d23a1a8899b35cc2138 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 311f765f59754bdcb4eadc04e14bd4ea, da3584f429104d23a1a8899b35cc2138; no voters: 
I20240502 01:54:18.625068 18828 raft_consensus.cc:2781] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:18.625177 18828 raft_consensus.cc:482] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:18.625322 18828 raft_consensus.cc:3037] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:18.627777 18828 raft_consensus.cc:504] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } }
I20240502 01:54:18.628479 18828 leader_election.cc:288] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P da3584f429104d23a1a8899b35cc2138 [CANDIDATE]: Term 1 election: Requested vote from peers 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193:35397), 84d3a4b174c34e02aedab1e620731359 (127.9.176.195:37357)
I20240502 01:54:18.628971 17498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60ff2e50b1d74e0c8e0890b87f9ac8ba" candidate_uuid: "da3584f429104d23a1a8899b35cc2138" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "311f765f59754bdcb4eadc04e14bd4ea"
I20240502 01:54:18.629007 17766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60ff2e50b1d74e0c8e0890b87f9ac8ba" candidate_uuid: "da3584f429104d23a1a8899b35cc2138" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84d3a4b174c34e02aedab1e620731359"
I20240502 01:54:18.629189 17498 raft_consensus.cc:3037] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:18.629225 17766 raft_consensus.cc:3037] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:18.631217 17766 raft_consensus.cc:2454] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da3584f429104d23a1a8899b35cc2138 in term 1.
I20240502 01:54:18.631223 17498 raft_consensus.cc:2454] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da3584f429104d23a1a8899b35cc2138 in term 1.
I20240502 01:54:18.631736 17835 leader_election.cc:302] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P da3584f429104d23a1a8899b35cc2138 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 311f765f59754bdcb4eadc04e14bd4ea, da3584f429104d23a1a8899b35cc2138; no voters: 
I20240502 01:54:18.632030 18828 raft_consensus.cc:2781] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:18.632508 18828 raft_consensus.cc:686] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P da3584f429104d23a1a8899b35cc2138 [term 1 LEADER]: Becoming Leader. State: Replica: da3584f429104d23a1a8899b35cc2138, State: Running, Role: LEADER
I20240502 01:54:18.632855 18828 consensus_queue.cc:237] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P da3584f429104d23a1a8899b35cc2138 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } }
I20240502 01:54:18.635862 17347 catalog_manager.cc:5484] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P da3584f429104d23a1a8899b35cc2138 reported cstate change: term changed from 0 to 1, leader changed from <none> to da3584f429104d23a1a8899b35cc2138 (127.9.176.196). New cstate: current_term: 1 leader_uuid: "da3584f429104d23a1a8899b35cc2138" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:18.974925 18972 consensus_queue.cc:1035] T a6b5231a18a5414dbd010d4d48f08dbd P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:18.984589 19042 consensus_queue.cc:1035] T a6b5231a18a5414dbd010d4d48f08dbd P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:19.046962 19042 consensus_queue.cc:1035] T 075ded9523e44c3d92844805338643a3 P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:19.052721 18972 consensus_queue.cc:1035] T 075ded9523e44c3d92844805338643a3 P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:54:19.054370 19065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:54:19.054864 19065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:54:19.066361 19065 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 01:54:19.098656 18828 consensus_queue.cc:1035] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P da3584f429104d23a1a8899b35cc2138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:19.099947 18828 consensus_queue.cc:1035] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P da3584f429104d23a1a8899b35cc2138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_6'

INSERT report
    rows total: 2000
    time total: 263.427 ms
  time per row: 0.131714 ms
I20240502 01:54:19.500581 17347 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56698:
name: "rebalance_test_table_7"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 01:54:19.509775 17747 tablet_service.cc:1467] Processing CreateTablet for tablet 2c78951695284b1aaab411c0a041c28a (DEFAULT_TABLE table=rebalance_test_table_7 [id=1bd69df0513c4c4db4dd6d6526efee4b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:19.509872 17479 tablet_service.cc:1467] Processing CreateTablet for tablet 2c78951695284b1aaab411c0a041c28a (DEFAULT_TABLE table=rebalance_test_table_7 [id=1bd69df0513c4c4db4dd6d6526efee4b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:19.510329 17747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c78951695284b1aaab411c0a041c28a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:19.510339 17479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c78951695284b1aaab411c0a041c28a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:19.511510 17477 tablet_service.cc:1467] Processing CreateTablet for tablet d022e878d5d943eb849e53d15029aa77 (DEFAULT_TABLE table=rebalance_test_table_7 [id=1bd69df0513c4c4db4dd6d6526efee4b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:19.512010 17477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d022e878d5d943eb849e53d15029aa77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:19.512346 17611 tablet_service.cc:1467] Processing CreateTablet for tablet 2c78951695284b1aaab411c0a041c28a (DEFAULT_TABLE table=rebalance_test_table_7 [id=1bd69df0513c4c4db4dd6d6526efee4b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:19.512786 17611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c78951695284b1aaab411c0a041c28a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:19.516544 17879 tablet_service.cc:1467] Processing CreateTablet for tablet d022e878d5d943eb849e53d15029aa77 (DEFAULT_TABLE table=rebalance_test_table_7 [id=1bd69df0513c4c4db4dd6d6526efee4b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:19.517010 17879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d022e878d5d943eb849e53d15029aa77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:19.517050 17745 tablet_service.cc:1467] Processing CreateTablet for tablet d022e878d5d943eb849e53d15029aa77 (DEFAULT_TABLE table=rebalance_test_table_7 [id=1bd69df0513c4c4db4dd6d6526efee4b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:19.517516 17612 tablet_service.cc:1467] Processing CreateTablet for tablet 5f3f3e5f772b4566b69236fd87d71a38 (DEFAULT_TABLE table=rebalance_test_table_7 [id=1bd69df0513c4c4db4dd6d6526efee4b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:19.517853 17612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f3f3e5f772b4566b69236fd87d71a38. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:19.521195 17745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d022e878d5d943eb849e53d15029aa77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:19.524708 17880 tablet_service.cc:1467] Processing CreateTablet for tablet 5f3f3e5f772b4566b69236fd87d71a38 (DEFAULT_TABLE table=rebalance_test_table_7 [id=1bd69df0513c4c4db4dd6d6526efee4b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:19.525171 17880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f3f3e5f772b4566b69236fd87d71a38. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:19.525090 17746 tablet_service.cc:1467] Processing CreateTablet for tablet 5f3f3e5f772b4566b69236fd87d71a38 (DEFAULT_TABLE table=rebalance_test_table_7 [id=1bd69df0513c4c4db4dd6d6526efee4b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:19.525532 17746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f3f3e5f772b4566b69236fd87d71a38. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:19.528378 19133 tablet_bootstrap.cc:492] T d022e878d5d943eb849e53d15029aa77 P da3584f429104d23a1a8899b35cc2138: Bootstrap starting.
I20240502 01:54:19.530798 19133 tablet_bootstrap.cc:654] T d022e878d5d943eb849e53d15029aa77 P da3584f429104d23a1a8899b35cc2138: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:19.533123 19131 tablet_bootstrap.cc:492] T 2c78951695284b1aaab411c0a041c28a P 1b50a031f2784ba498b3e52e16916205: Bootstrap starting.
I20240502 01:54:19.535588 19131 tablet_bootstrap.cc:654] T 2c78951695284b1aaab411c0a041c28a P 1b50a031f2784ba498b3e52e16916205: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:19.536592 19132 tablet_bootstrap.cc:492] T 2c78951695284b1aaab411c0a041c28a P 311f765f59754bdcb4eadc04e14bd4ea: Bootstrap starting.
I20240502 01:54:19.536615 19130 tablet_bootstrap.cc:492] T 2c78951695284b1aaab411c0a041c28a P 84d3a4b174c34e02aedab1e620731359: Bootstrap starting.
I20240502 01:54:19.539115 19132 tablet_bootstrap.cc:654] T 2c78951695284b1aaab411c0a041c28a P 311f765f59754bdcb4eadc04e14bd4ea: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:19.539669 19130 tablet_bootstrap.cc:654] T 2c78951695284b1aaab411c0a041c28a P 84d3a4b174c34e02aedab1e620731359: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:19.541553 19131 tablet_bootstrap.cc:492] T 2c78951695284b1aaab411c0a041c28a P 1b50a031f2784ba498b3e52e16916205: No bootstrap required, opened a new log
I20240502 01:54:19.541730 19131 ts_tablet_manager.cc:1367] T 2c78951695284b1aaab411c0a041c28a P 1b50a031f2784ba498b3e52e16916205: Time spent bootstrapping tablet: real 0.009s	user 0.000s	sys 0.007s
I20240502 01:54:19.543396 19133 tablet_bootstrap.cc:492] T d022e878d5d943eb849e53d15029aa77 P da3584f429104d23a1a8899b35cc2138: No bootstrap required, opened a new log
I20240502 01:54:19.543816 19131 raft_consensus.cc:348] T 2c78951695284b1aaab411c0a041c28a P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } }
I20240502 01:54:19.545923 19131 raft_consensus.cc:374] T 2c78951695284b1aaab411c0a041c28a P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:19.546082 19131 raft_consensus.cc:729] T 2c78951695284b1aaab411c0a041c28a P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b50a031f2784ba498b3e52e16916205, State: Initialized, Role: FOLLOWER
I20240502 01:54:19.546581 19131 consensus_queue.cc:260] T 2c78951695284b1aaab411c0a041c28a P 1b50a031f2784ba498b3e52e16916205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } }
I20240502 01:54:19.547329 19133 ts_tablet_manager.cc:1367] T d022e878d5d943eb849e53d15029aa77 P da3584f429104d23a1a8899b35cc2138: Time spent bootstrapping tablet: real 0.019s	user 0.011s	sys 0.000s
I20240502 01:54:19.547894 19131 ts_tablet_manager.cc:1398] T 2c78951695284b1aaab411c0a041c28a P 1b50a031f2784ba498b3e52e16916205: Time spent starting tablet: real 0.006s	user 0.000s	sys 0.003s
I20240502 01:54:19.548194 19131 tablet_bootstrap.cc:492] T 5f3f3e5f772b4566b69236fd87d71a38 P 1b50a031f2784ba498b3e52e16916205: Bootstrap starting.
I20240502 01:54:19.548456 19132 tablet_bootstrap.cc:492] T 2c78951695284b1aaab411c0a041c28a P 311f765f59754bdcb4eadc04e14bd4ea: No bootstrap required, opened a new log
I20240502 01:54:19.548666 19132 ts_tablet_manager.cc:1367] T 2c78951695284b1aaab411c0a041c28a P 311f765f59754bdcb4eadc04e14bd4ea: Time spent bootstrapping tablet: real 0.013s	user 0.005s	sys 0.005s
I20240502 01:54:19.548813 19133 raft_consensus.cc:348] T d022e878d5d943eb849e53d15029aa77 P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } }
I20240502 01:54:19.549160 19133 raft_consensus.cc:374] T d022e878d5d943eb849e53d15029aa77 P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:19.549299 19133 raft_consensus.cc:729] T d022e878d5d943eb849e53d15029aa77 P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da3584f429104d23a1a8899b35cc2138, State: Initialized, Role: FOLLOWER
I20240502 01:54:19.549600 19132 raft_consensus.cc:348] T 2c78951695284b1aaab411c0a041c28a P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } }
I20240502 01:54:19.549816 19132 raft_consensus.cc:374] T 2c78951695284b1aaab411c0a041c28a P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:19.549657 19133 consensus_queue.cc:260] T d022e878d5d943eb849e53d15029aa77 P da3584f429104d23a1a8899b35cc2138 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } }
I20240502 01:54:19.549917 19132 raft_consensus.cc:729] T 2c78951695284b1aaab411c0a041c28a P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 311f765f59754bdcb4eadc04e14bd4ea, State: Initialized, Role: FOLLOWER
I20240502 01:54:19.549993 19131 tablet_bootstrap.cc:654] T 5f3f3e5f772b4566b69236fd87d71a38 P 1b50a031f2784ba498b3e52e16916205: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:19.550176 19132 consensus_queue.cc:260] T 2c78951695284b1aaab411c0a041c28a P 311f765f59754bdcb4eadc04e14bd4ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } }
I20240502 01:54:19.550637 19133 ts_tablet_manager.cc:1398] T d022e878d5d943eb849e53d15029aa77 P da3584f429104d23a1a8899b35cc2138: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 01:54:19.550961 19133 tablet_bootstrap.cc:492] T 5f3f3e5f772b4566b69236fd87d71a38 P da3584f429104d23a1a8899b35cc2138: Bootstrap starting.
I20240502 01:54:19.551692 19132 ts_tablet_manager.cc:1398] T 2c78951695284b1aaab411c0a041c28a P 311f765f59754bdcb4eadc04e14bd4ea: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.001s
I20240502 01:54:19.552085 19132 tablet_bootstrap.cc:492] T d022e878d5d943eb849e53d15029aa77 P 311f765f59754bdcb4eadc04e14bd4ea: Bootstrap starting.
I20240502 01:54:19.554983 19132 tablet_bootstrap.cc:654] T d022e878d5d943eb849e53d15029aa77 P 311f765f59754bdcb4eadc04e14bd4ea: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:19.562142 19133 tablet_bootstrap.cc:654] T 5f3f3e5f772b4566b69236fd87d71a38 P da3584f429104d23a1a8899b35cc2138: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:19.568207 19133 tablet_bootstrap.cc:492] T 5f3f3e5f772b4566b69236fd87d71a38 P da3584f429104d23a1a8899b35cc2138: No bootstrap required, opened a new log
I20240502 01:54:19.568603 19133 ts_tablet_manager.cc:1367] T 5f3f3e5f772b4566b69236fd87d71a38 P da3584f429104d23a1a8899b35cc2138: Time spent bootstrapping tablet: real 0.018s	user 0.005s	sys 0.001s
I20240502 01:54:19.569836 19133 raft_consensus.cc:348] T 5f3f3e5f772b4566b69236fd87d71a38 P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } }
I20240502 01:54:19.570294 19133 raft_consensus.cc:374] T 5f3f3e5f772b4566b69236fd87d71a38 P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:19.570485 19133 raft_consensus.cc:729] T 5f3f3e5f772b4566b69236fd87d71a38 P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da3584f429104d23a1a8899b35cc2138, State: Initialized, Role: FOLLOWER
I20240502 01:54:19.571439 19133 consensus_queue.cc:260] T 5f3f3e5f772b4566b69236fd87d71a38 P da3584f429104d23a1a8899b35cc2138 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } }
I20240502 01:54:19.571750 19130 tablet_bootstrap.cc:492] T 2c78951695284b1aaab411c0a041c28a P 84d3a4b174c34e02aedab1e620731359: No bootstrap required, opened a new log
I20240502 01:54:19.571908 19130 ts_tablet_manager.cc:1367] T 2c78951695284b1aaab411c0a041c28a P 84d3a4b174c34e02aedab1e620731359: Time spent bootstrapping tablet: real 0.035s	user 0.010s	sys 0.000s
I20240502 01:54:19.572170 19131 tablet_bootstrap.cc:492] T 5f3f3e5f772b4566b69236fd87d71a38 P 1b50a031f2784ba498b3e52e16916205: No bootstrap required, opened a new log
I20240502 01:54:19.572242 19133 ts_tablet_manager.cc:1398] T 5f3f3e5f772b4566b69236fd87d71a38 P da3584f429104d23a1a8899b35cc2138: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.002s
I20240502 01:54:19.572355 19131 ts_tablet_manager.cc:1367] T 5f3f3e5f772b4566b69236fd87d71a38 P 1b50a031f2784ba498b3e52e16916205: Time spent bootstrapping tablet: real 0.024s	user 0.009s	sys 0.013s
I20240502 01:54:19.572786 19130 raft_consensus.cc:348] T 2c78951695284b1aaab411c0a041c28a P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } }
I20240502 01:54:19.572999 19130 raft_consensus.cc:374] T 2c78951695284b1aaab411c0a041c28a P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:19.573035 19132 tablet_bootstrap.cc:492] T d022e878d5d943eb849e53d15029aa77 P 311f765f59754bdcb4eadc04e14bd4ea: No bootstrap required, opened a new log
I20240502 01:54:19.573098 19130 raft_consensus.cc:729] T 2c78951695284b1aaab411c0a041c28a P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84d3a4b174c34e02aedab1e620731359, State: Initialized, Role: FOLLOWER
I20240502 01:54:19.573217 19132 ts_tablet_manager.cc:1367] T d022e878d5d943eb849e53d15029aa77 P 311f765f59754bdcb4eadc04e14bd4ea: Time spent bootstrapping tablet: real 0.021s	user 0.018s	sys 0.000s
I20240502 01:54:19.573350 19130 consensus_queue.cc:260] T 2c78951695284b1aaab411c0a041c28a P 84d3a4b174c34e02aedab1e620731359 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } }
I20240502 01:54:19.573191 19131 raft_consensus.cc:348] T 5f3f3e5f772b4566b69236fd87d71a38 P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } }
I20240502 01:54:19.573781 19131 raft_consensus.cc:374] T 5f3f3e5f772b4566b69236fd87d71a38 P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:19.573894 19131 raft_consensus.cc:729] T 5f3f3e5f772b4566b69236fd87d71a38 P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b50a031f2784ba498b3e52e16916205, State: Initialized, Role: FOLLOWER
I20240502 01:54:19.573925 19130 ts_tablet_manager.cc:1398] T 2c78951695284b1aaab411c0a041c28a P 84d3a4b174c34e02aedab1e620731359: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 01:54:19.574033 19132 raft_consensus.cc:348] T d022e878d5d943eb849e53d15029aa77 P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } }
I20240502 01:54:19.574268 19130 tablet_bootstrap.cc:492] T d022e878d5d943eb849e53d15029aa77 P 84d3a4b174c34e02aedab1e620731359: Bootstrap starting.
I20240502 01:54:19.574267 19132 raft_consensus.cc:374] T d022e878d5d943eb849e53d15029aa77 P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:19.574188 19131 consensus_queue.cc:260] T 5f3f3e5f772b4566b69236fd87d71a38 P 1b50a031f2784ba498b3e52e16916205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } }
I20240502 01:54:19.574347 19132 raft_consensus.cc:729] T d022e878d5d943eb849e53d15029aa77 P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 311f765f59754bdcb4eadc04e14bd4ea, State: Initialized, Role: FOLLOWER
I20240502 01:54:19.574607 19132 consensus_queue.cc:260] T d022e878d5d943eb849e53d15029aa77 P 311f765f59754bdcb4eadc04e14bd4ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } }
I20240502 01:54:19.574918 19131 ts_tablet_manager.cc:1398] T 5f3f3e5f772b4566b69236fd87d71a38 P 1b50a031f2784ba498b3e52e16916205: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.000s
I20240502 01:54:19.576122 19132 ts_tablet_manager.cc:1398] T d022e878d5d943eb849e53d15029aa77 P 311f765f59754bdcb4eadc04e14bd4ea: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 01:54:19.577678 19130 tablet_bootstrap.cc:654] T d022e878d5d943eb849e53d15029aa77 P 84d3a4b174c34e02aedab1e620731359: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:19.579918 19130 tablet_bootstrap.cc:492] T d022e878d5d943eb849e53d15029aa77 P 84d3a4b174c34e02aedab1e620731359: No bootstrap required, opened a new log
I20240502 01:54:19.580096 19130 ts_tablet_manager.cc:1367] T d022e878d5d943eb849e53d15029aa77 P 84d3a4b174c34e02aedab1e620731359: Time spent bootstrapping tablet: real 0.006s	user 0.000s	sys 0.003s
I20240502 01:54:19.580770 19130 raft_consensus.cc:348] T d022e878d5d943eb849e53d15029aa77 P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } }
I20240502 01:54:19.580981 19130 raft_consensus.cc:374] T d022e878d5d943eb849e53d15029aa77 P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:19.581063 19130 raft_consensus.cc:729] T d022e878d5d943eb849e53d15029aa77 P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84d3a4b174c34e02aedab1e620731359, State: Initialized, Role: FOLLOWER
I20240502 01:54:19.581326 19130 consensus_queue.cc:260] T d022e878d5d943eb849e53d15029aa77 P 84d3a4b174c34e02aedab1e620731359 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } }
I20240502 01:54:19.581902 19130 ts_tablet_manager.cc:1398] T d022e878d5d943eb849e53d15029aa77 P 84d3a4b174c34e02aedab1e620731359: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 01:54:19.582276 19130 tablet_bootstrap.cc:492] T 5f3f3e5f772b4566b69236fd87d71a38 P 84d3a4b174c34e02aedab1e620731359: Bootstrap starting.
I20240502 01:54:19.583732 19130 tablet_bootstrap.cc:654] T 5f3f3e5f772b4566b69236fd87d71a38 P 84d3a4b174c34e02aedab1e620731359: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:19.586072 19130 tablet_bootstrap.cc:492] T 5f3f3e5f772b4566b69236fd87d71a38 P 84d3a4b174c34e02aedab1e620731359: No bootstrap required, opened a new log
I20240502 01:54:19.586221 19130 ts_tablet_manager.cc:1367] T 5f3f3e5f772b4566b69236fd87d71a38 P 84d3a4b174c34e02aedab1e620731359: Time spent bootstrapping tablet: real 0.004s	user 0.000s	sys 0.003s
I20240502 01:54:19.586796 19130 raft_consensus.cc:348] T 5f3f3e5f772b4566b69236fd87d71a38 P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } }
I20240502 01:54:19.586953 19130 raft_consensus.cc:374] T 5f3f3e5f772b4566b69236fd87d71a38 P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:19.587020 19130 raft_consensus.cc:729] T 5f3f3e5f772b4566b69236fd87d71a38 P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84d3a4b174c34e02aedab1e620731359, State: Initialized, Role: FOLLOWER
I20240502 01:54:19.587281 19130 consensus_queue.cc:260] T 5f3f3e5f772b4566b69236fd87d71a38 P 84d3a4b174c34e02aedab1e620731359 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } }
I20240502 01:54:19.587908 19130 ts_tablet_manager.cc:1398] T 5f3f3e5f772b4566b69236fd87d71a38 P 84d3a4b174c34e02aedab1e620731359: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240502 01:54:19.665915 18989 raft_consensus.cc:482] T 5f3f3e5f772b4566b69236fd87d71a38 P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:19.666158 18989 raft_consensus.cc:504] T 5f3f3e5f772b4566b69236fd87d71a38 P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } }
I20240502 01:54:19.667028 18989 leader_election.cc:288] T 5f3f3e5f772b4566b69236fd87d71a38 P 1b50a031f2784ba498b3e52e16916205 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers da3584f429104d23a1a8899b35cc2138 (127.9.176.196:46381), 84d3a4b174c34e02aedab1e620731359 (127.9.176.195:37357)
I20240502 01:54:19.667409 17899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f3f3e5f772b4566b69236fd87d71a38" candidate_uuid: "1b50a031f2784ba498b3e52e16916205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da3584f429104d23a1a8899b35cc2138" is_pre_election: true
I20240502 01:54:19.667560 17766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f3f3e5f772b4566b69236fd87d71a38" candidate_uuid: "1b50a031f2784ba498b3e52e16916205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84d3a4b174c34e02aedab1e620731359" is_pre_election: true
I20240502 01:54:19.667739 17899 raft_consensus.cc:2454] T 5f3f3e5f772b4566b69236fd87d71a38 P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b50a031f2784ba498b3e52e16916205 in term 0.
I20240502 01:54:19.667809 17766 raft_consensus.cc:2454] T 5f3f3e5f772b4566b69236fd87d71a38 P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b50a031f2784ba498b3e52e16916205 in term 0.
I20240502 01:54:19.668237 17565 leader_election.cc:302] T 5f3f3e5f772b4566b69236fd87d71a38 P 1b50a031f2784ba498b3e52e16916205 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1b50a031f2784ba498b3e52e16916205, 84d3a4b174c34e02aedab1e620731359, da3584f429104d23a1a8899b35cc2138; no voters: 
I20240502 01:54:19.668567 18989 raft_consensus.cc:2781] T 5f3f3e5f772b4566b69236fd87d71a38 P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:19.668689 18989 raft_consensus.cc:482] T 5f3f3e5f772b4566b69236fd87d71a38 P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:19.668783 18989 raft_consensus.cc:3037] T 5f3f3e5f772b4566b69236fd87d71a38 P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:19.671007 18989 raft_consensus.cc:504] T 5f3f3e5f772b4566b69236fd87d71a38 P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } }
I20240502 01:54:19.671648 18989 leader_election.cc:288] T 5f3f3e5f772b4566b69236fd87d71a38 P 1b50a031f2784ba498b3e52e16916205 [CANDIDATE]: Term 1 election: Requested vote from peers da3584f429104d23a1a8899b35cc2138 (127.9.176.196:46381), 84d3a4b174c34e02aedab1e620731359 (127.9.176.195:37357)
I20240502 01:54:19.672133 17899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f3f3e5f772b4566b69236fd87d71a38" candidate_uuid: "1b50a031f2784ba498b3e52e16916205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da3584f429104d23a1a8899b35cc2138"
I20240502 01:54:19.672360 17899 raft_consensus.cc:3037] T 5f3f3e5f772b4566b69236fd87d71a38 P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:19.672374 17766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f3f3e5f772b4566b69236fd87d71a38" candidate_uuid: "1b50a031f2784ba498b3e52e16916205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84d3a4b174c34e02aedab1e620731359"
I20240502 01:54:19.672576 17766 raft_consensus.cc:3037] T 5f3f3e5f772b4566b69236fd87d71a38 P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:19.675351 17899 raft_consensus.cc:2454] T 5f3f3e5f772b4566b69236fd87d71a38 P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b50a031f2784ba498b3e52e16916205 in term 1.
I20240502 01:54:19.675351 17766 raft_consensus.cc:2454] T 5f3f3e5f772b4566b69236fd87d71a38 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b50a031f2784ba498b3e52e16916205 in term 1.
I20240502 01:54:19.675917 17565 leader_election.cc:302] T 5f3f3e5f772b4566b69236fd87d71a38 P 1b50a031f2784ba498b3e52e16916205 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b50a031f2784ba498b3e52e16916205, da3584f429104d23a1a8899b35cc2138; no voters: 
I20240502 01:54:19.676311 18989 raft_consensus.cc:2781] T 5f3f3e5f772b4566b69236fd87d71a38 P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:19.676728 18989 raft_consensus.cc:686] T 5f3f3e5f772b4566b69236fd87d71a38 P 1b50a031f2784ba498b3e52e16916205 [term 1 LEADER]: Becoming Leader. State: Replica: 1b50a031f2784ba498b3e52e16916205, State: Running, Role: LEADER
I20240502 01:54:19.677126 18989 consensus_queue.cc:237] T 5f3f3e5f772b4566b69236fd87d71a38 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } }
I20240502 01:54:19.680347 17347 catalog_manager.cc:5484] T 5f3f3e5f772b4566b69236fd87d71a38 P 1b50a031f2784ba498b3e52e16916205 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b50a031f2784ba498b3e52e16916205 (127.9.176.194). New cstate: current_term: 1 leader_uuid: "1b50a031f2784ba498b3e52e16916205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:19.712114 18989 raft_consensus.cc:482] T 2c78951695284b1aaab411c0a041c28a P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:19.712363 18989 raft_consensus.cc:504] T 2c78951695284b1aaab411c0a041c28a P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } }
I20240502 01:54:19.713045 18989 leader_election.cc:288] T 2c78951695284b1aaab411c0a041c28a P 1b50a031f2784ba498b3e52e16916205 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84d3a4b174c34e02aedab1e620731359 (127.9.176.195:37357), 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193:35397)
I20240502 01:54:19.713527 17766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c78951695284b1aaab411c0a041c28a" candidate_uuid: "1b50a031f2784ba498b3e52e16916205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84d3a4b174c34e02aedab1e620731359" is_pre_election: true
I20240502 01:54:19.713578 17498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c78951695284b1aaab411c0a041c28a" candidate_uuid: "1b50a031f2784ba498b3e52e16916205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "311f765f59754bdcb4eadc04e14bd4ea" is_pre_election: true
I20240502 01:54:19.713778 17766 raft_consensus.cc:2454] T 2c78951695284b1aaab411c0a041c28a P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b50a031f2784ba498b3e52e16916205 in term 0.
I20240502 01:54:19.713800 17498 raft_consensus.cc:2454] T 2c78951695284b1aaab411c0a041c28a P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b50a031f2784ba498b3e52e16916205 in term 0.
I20240502 01:54:19.714205 17568 leader_election.cc:302] T 2c78951695284b1aaab411c0a041c28a P 1b50a031f2784ba498b3e52e16916205 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b50a031f2784ba498b3e52e16916205, 311f765f59754bdcb4eadc04e14bd4ea; no voters: 
I20240502 01:54:19.714511 18989 raft_consensus.cc:2781] T 2c78951695284b1aaab411c0a041c28a P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:19.714619 18989 raft_consensus.cc:482] T 2c78951695284b1aaab411c0a041c28a P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:19.714731 18989 raft_consensus.cc:3037] T 2c78951695284b1aaab411c0a041c28a P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:19.716684 18989 raft_consensus.cc:504] T 2c78951695284b1aaab411c0a041c28a P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } }
I20240502 01:54:19.717308 18989 leader_election.cc:288] T 2c78951695284b1aaab411c0a041c28a P 1b50a031f2784ba498b3e52e16916205 [CANDIDATE]: Term 1 election: Requested vote from peers 84d3a4b174c34e02aedab1e620731359 (127.9.176.195:37357), 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193:35397)
I20240502 01:54:19.717854 17498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c78951695284b1aaab411c0a041c28a" candidate_uuid: "1b50a031f2784ba498b3e52e16916205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "311f765f59754bdcb4eadc04e14bd4ea"
I20240502 01:54:19.718034 17766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c78951695284b1aaab411c0a041c28a" candidate_uuid: "1b50a031f2784ba498b3e52e16916205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84d3a4b174c34e02aedab1e620731359"
I20240502 01:54:19.718120 17498 raft_consensus.cc:3037] T 2c78951695284b1aaab411c0a041c28a P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:19.718261 17766 raft_consensus.cc:3037] T 2c78951695284b1aaab411c0a041c28a P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:19.721307 17766 raft_consensus.cc:2454] T 2c78951695284b1aaab411c0a041c28a P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b50a031f2784ba498b3e52e16916205 in term 1.
I20240502 01:54:19.721324 17498 raft_consensus.cc:2454] T 2c78951695284b1aaab411c0a041c28a P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b50a031f2784ba498b3e52e16916205 in term 1.
I20240502 01:54:19.722479 17568 leader_election.cc:302] T 2c78951695284b1aaab411c0a041c28a P 1b50a031f2784ba498b3e52e16916205 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b50a031f2784ba498b3e52e16916205, 311f765f59754bdcb4eadc04e14bd4ea; no voters: 
I20240502 01:54:19.723284 19143 raft_consensus.cc:2781] T 2c78951695284b1aaab411c0a041c28a P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:19.724959 19143 raft_consensus.cc:686] T 2c78951695284b1aaab411c0a041c28a P 1b50a031f2784ba498b3e52e16916205 [term 1 LEADER]: Becoming Leader. State: Replica: 1b50a031f2784ba498b3e52e16916205, State: Running, Role: LEADER
I20240502 01:54:19.725292 19143 consensus_queue.cc:237] T 2c78951695284b1aaab411c0a041c28a P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } }
I20240502 01:54:19.730022 17347 catalog_manager.cc:5484] T 2c78951695284b1aaab411c0a041c28a P 1b50a031f2784ba498b3e52e16916205 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b50a031f2784ba498b3e52e16916205 (127.9.176.194). New cstate: current_term: 1 leader_uuid: "1b50a031f2784ba498b3e52e16916205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:19.731484 18954 raft_consensus.cc:482] T d022e878d5d943eb849e53d15029aa77 P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:19.731707 18954 raft_consensus.cc:504] T d022e878d5d943eb849e53d15029aa77 P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } }
I20240502 01:54:19.732569 18954 leader_election.cc:288] T d022e878d5d943eb849e53d15029aa77 P 84d3a4b174c34e02aedab1e620731359 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193:35397), da3584f429104d23a1a8899b35cc2138 (127.9.176.196:46381)
I20240502 01:54:19.733230 17899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d022e878d5d943eb849e53d15029aa77" candidate_uuid: "84d3a4b174c34e02aedab1e620731359" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da3584f429104d23a1a8899b35cc2138" is_pre_election: true
I20240502 01:54:19.733377 17498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d022e878d5d943eb849e53d15029aa77" candidate_uuid: "84d3a4b174c34e02aedab1e620731359" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "311f765f59754bdcb4eadc04e14bd4ea" is_pre_election: true
I20240502 01:54:19.733513 17899 raft_consensus.cc:2454] T d022e878d5d943eb849e53d15029aa77 P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84d3a4b174c34e02aedab1e620731359 in term 0.
I20240502 01:54:19.733649 17498 raft_consensus.cc:2454] T d022e878d5d943eb849e53d15029aa77 P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84d3a4b174c34e02aedab1e620731359 in term 0.
I20240502 01:54:19.734143 17703 leader_election.cc:302] T d022e878d5d943eb849e53d15029aa77 P 84d3a4b174c34e02aedab1e620731359 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 311f765f59754bdcb4eadc04e14bd4ea, 84d3a4b174c34e02aedab1e620731359; no voters: 
I20240502 01:54:19.734447 18954 raft_consensus.cc:2781] T d022e878d5d943eb849e53d15029aa77 P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:19.734575 18954 raft_consensus.cc:482] T d022e878d5d943eb849e53d15029aa77 P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:19.734670 18954 raft_consensus.cc:3037] T d022e878d5d943eb849e53d15029aa77 P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:19.737579 18954 raft_consensus.cc:504] T d022e878d5d943eb849e53d15029aa77 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } }
I20240502 01:54:19.738157 18954 leader_election.cc:288] T d022e878d5d943eb849e53d15029aa77 P 84d3a4b174c34e02aedab1e620731359 [CANDIDATE]: Term 1 election: Requested vote from peers 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193:35397), da3584f429104d23a1a8899b35cc2138 (127.9.176.196:46381)
I20240502 01:54:19.738693 17899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d022e878d5d943eb849e53d15029aa77" candidate_uuid: "84d3a4b174c34e02aedab1e620731359" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da3584f429104d23a1a8899b35cc2138"
I20240502 01:54:19.738937 17899 raft_consensus.cc:3037] T d022e878d5d943eb849e53d15029aa77 P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:19.738951 17498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d022e878d5d943eb849e53d15029aa77" candidate_uuid: "84d3a4b174c34e02aedab1e620731359" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "311f765f59754bdcb4eadc04e14bd4ea"
I20240502 01:54:19.739248 17498 raft_consensus.cc:3037] T d022e878d5d943eb849e53d15029aa77 P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:19.741441 17899 raft_consensus.cc:2454] T d022e878d5d943eb849e53d15029aa77 P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84d3a4b174c34e02aedab1e620731359 in term 1.
I20240502 01:54:19.741900 17700 leader_election.cc:302] T d022e878d5d943eb849e53d15029aa77 P 84d3a4b174c34e02aedab1e620731359 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84d3a4b174c34e02aedab1e620731359, da3584f429104d23a1a8899b35cc2138; no voters: 
I20240502 01:54:19.742189 18954 raft_consensus.cc:2781] T d022e878d5d943eb849e53d15029aa77 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:19.742202 17498 raft_consensus.cc:2454] T d022e878d5d943eb849e53d15029aa77 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84d3a4b174c34e02aedab1e620731359 in term 1.
I20240502 01:54:19.742745 18954 raft_consensus.cc:686] T d022e878d5d943eb849e53d15029aa77 P 84d3a4b174c34e02aedab1e620731359 [term 1 LEADER]: Becoming Leader. State: Replica: 84d3a4b174c34e02aedab1e620731359, State: Running, Role: LEADER
I20240502 01:54:19.743052 18954 consensus_queue.cc:237] T d022e878d5d943eb849e53d15029aa77 P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } }
I20240502 01:54:19.747882 17347 catalog_manager.cc:5484] T d022e878d5d943eb849e53d15029aa77 P 84d3a4b174c34e02aedab1e620731359 reported cstate change: term changed from 0 to 1, leader changed from <none> to 84d3a4b174c34e02aedab1e620731359 (127.9.176.195). New cstate: current_term: 1 leader_uuid: "84d3a4b174c34e02aedab1e620731359" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:20.153066 18954 consensus_queue.cc:1035] T d022e878d5d943eb849e53d15029aa77 P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:54:20.166841 19148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:54:20.167412 19148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240502 01:54:20.169991 19143 consensus_queue.cc:1035] T 2c78951695284b1aaab411c0a041c28a P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:54:20.180714 19148 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 01:54:20.184763 19120 consensus_queue.cc:1035] T d022e878d5d943eb849e53d15029aa77 P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:20.190780 19143 consensus_queue.cc:1035] T 2c78951695284b1aaab411c0a041c28a P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:20.211951 19143 consensus_queue.cc:1035] T 5f3f3e5f772b4566b69236fd87d71a38 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:20.220026 19145 consensus_queue.cc:1035] T 5f3f3e5f772b4566b69236fd87d71a38 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_7'

INSERT report
    rows total: 2000
    time total: 254.757 ms
  time per row: 0.127379 ms
I20240502 01:54:20.638543 17354 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56698:
name: "rebalance_test_table_8"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 01:54:20.649356 17611 tablet_service.cc:1467] Processing CreateTablet for tablet df16e178e04040699201f325ce9e2979 (DEFAULT_TABLE table=rebalance_test_table_8 [id=40a6be2c1a2744428cea56b78225e44e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:20.649425 17479 tablet_service.cc:1467] Processing CreateTablet for tablet df16e178e04040699201f325ce9e2979 (DEFAULT_TABLE table=rebalance_test_table_8 [id=40a6be2c1a2744428cea56b78225e44e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:20.649892 17611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df16e178e04040699201f325ce9e2979. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:20.649927 17479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df16e178e04040699201f325ce9e2979. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:20.650159 17879 tablet_service.cc:1467] Processing CreateTablet for tablet 49ef898a43704b87ba077cd7b0485e9a (DEFAULT_TABLE table=rebalance_test_table_8 [id=40a6be2c1a2744428cea56b78225e44e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:20.650583 17879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49ef898a43704b87ba077cd7b0485e9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:20.651448 17477 tablet_service.cc:1467] Processing CreateTablet for tablet 49ef898a43704b87ba077cd7b0485e9a (DEFAULT_TABLE table=rebalance_test_table_8 [id=40a6be2c1a2744428cea56b78225e44e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:20.651891 17477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49ef898a43704b87ba077cd7b0485e9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:20.654596 17478 tablet_service.cc:1467] Processing CreateTablet for tablet fadcc8fe6ae047c1b142514aad8bebba (DEFAULT_TABLE table=rebalance_test_table_8 [id=40a6be2c1a2744428cea56b78225e44e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:20.655094 17478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fadcc8fe6ae047c1b142514aad8bebba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:20.655490 17747 tablet_service.cc:1467] Processing CreateTablet for tablet df16e178e04040699201f325ce9e2979 (DEFAULT_TABLE table=rebalance_test_table_8 [id=40a6be2c1a2744428cea56b78225e44e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:20.655651 17746 tablet_service.cc:1467] Processing CreateTablet for tablet fadcc8fe6ae047c1b142514aad8bebba (DEFAULT_TABLE table=rebalance_test_table_8 [id=40a6be2c1a2744428cea56b78225e44e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:20.655983 17747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df16e178e04040699201f325ce9e2979. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:20.656194 17746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fadcc8fe6ae047c1b142514aad8bebba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:20.660004 17880 tablet_service.cc:1467] Processing CreateTablet for tablet fadcc8fe6ae047c1b142514aad8bebba (DEFAULT_TABLE table=rebalance_test_table_8 [id=40a6be2c1a2744428cea56b78225e44e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:20.660475 17880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fadcc8fe6ae047c1b142514aad8bebba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:20.660748 17612 tablet_service.cc:1467] Processing CreateTablet for tablet 49ef898a43704b87ba077cd7b0485e9a (DEFAULT_TABLE table=rebalance_test_table_8 [id=40a6be2c1a2744428cea56b78225e44e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:20.661197 17612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49ef898a43704b87ba077cd7b0485e9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:20.667183 19212 tablet_bootstrap.cc:492] T 49ef898a43704b87ba077cd7b0485e9a P 311f765f59754bdcb4eadc04e14bd4ea: Bootstrap starting.
I20240502 01:54:20.669667 19212 tablet_bootstrap.cc:654] T 49ef898a43704b87ba077cd7b0485e9a P 311f765f59754bdcb4eadc04e14bd4ea: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:20.671504 19212 tablet_bootstrap.cc:492] T 49ef898a43704b87ba077cd7b0485e9a P 311f765f59754bdcb4eadc04e14bd4ea: No bootstrap required, opened a new log
I20240502 01:54:20.672040 19212 ts_tablet_manager.cc:1367] T 49ef898a43704b87ba077cd7b0485e9a P 311f765f59754bdcb4eadc04e14bd4ea: Time spent bootstrapping tablet: real 0.005s	user 0.004s	sys 0.000s
I20240502 01:54:20.672950 19212 raft_consensus.cc:348] T 49ef898a43704b87ba077cd7b0485e9a P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } }
I20240502 01:54:20.673380 19212 raft_consensus.cc:374] T 49ef898a43704b87ba077cd7b0485e9a P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:20.673486 19212 raft_consensus.cc:729] T 49ef898a43704b87ba077cd7b0485e9a P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 311f765f59754bdcb4eadc04e14bd4ea, State: Initialized, Role: FOLLOWER
I20240502 01:54:20.673718 19212 consensus_queue.cc:260] T 49ef898a43704b87ba077cd7b0485e9a P 311f765f59754bdcb4eadc04e14bd4ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } }
I20240502 01:54:20.674402 19212 ts_tablet_manager.cc:1398] T 49ef898a43704b87ba077cd7b0485e9a P 311f765f59754bdcb4eadc04e14bd4ea: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 01:54:20.674731 19212 tablet_bootstrap.cc:492] T df16e178e04040699201f325ce9e2979 P 311f765f59754bdcb4eadc04e14bd4ea: Bootstrap starting.
I20240502 01:54:20.676436 19212 tablet_bootstrap.cc:654] T df16e178e04040699201f325ce9e2979 P 311f765f59754bdcb4eadc04e14bd4ea: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:20.681962 19213 tablet_bootstrap.cc:492] T df16e178e04040699201f325ce9e2979 P 1b50a031f2784ba498b3e52e16916205: Bootstrap starting.
I20240502 01:54:20.682843 19212 tablet_bootstrap.cc:492] T df16e178e04040699201f325ce9e2979 P 311f765f59754bdcb4eadc04e14bd4ea: No bootstrap required, opened a new log
I20240502 01:54:20.682978 19214 tablet_bootstrap.cc:492] T df16e178e04040699201f325ce9e2979 P 84d3a4b174c34e02aedab1e620731359: Bootstrap starting.
I20240502 01:54:20.683041 19212 ts_tablet_manager.cc:1367] T df16e178e04040699201f325ce9e2979 P 311f765f59754bdcb4eadc04e14bd4ea: Time spent bootstrapping tablet: real 0.008s	user 0.007s	sys 0.000s
I20240502 01:54:20.683655 19216 tablet_bootstrap.cc:492] T 49ef898a43704b87ba077cd7b0485e9a P da3584f429104d23a1a8899b35cc2138: Bootstrap starting.
I20240502 01:54:20.683885 19212 raft_consensus.cc:348] T df16e178e04040699201f325ce9e2979 P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } }
I20240502 01:54:20.684101 19212 raft_consensus.cc:374] T df16e178e04040699201f325ce9e2979 P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:20.684186 19212 raft_consensus.cc:729] T df16e178e04040699201f325ce9e2979 P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 311f765f59754bdcb4eadc04e14bd4ea, State: Initialized, Role: FOLLOWER
I20240502 01:54:20.684481 19212 consensus_queue.cc:260] T df16e178e04040699201f325ce9e2979 P 311f765f59754bdcb4eadc04e14bd4ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } }
I20240502 01:54:20.684831 19213 tablet_bootstrap.cc:654] T df16e178e04040699201f325ce9e2979 P 1b50a031f2784ba498b3e52e16916205: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:20.685081 19212 ts_tablet_manager.cc:1398] T df16e178e04040699201f325ce9e2979 P 311f765f59754bdcb4eadc04e14bd4ea: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 01:54:20.685464 19214 tablet_bootstrap.cc:654] T df16e178e04040699201f325ce9e2979 P 84d3a4b174c34e02aedab1e620731359: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:20.685541 19212 tablet_bootstrap.cc:492] T fadcc8fe6ae047c1b142514aad8bebba P 311f765f59754bdcb4eadc04e14bd4ea: Bootstrap starting.
I20240502 01:54:20.687871 19212 tablet_bootstrap.cc:654] T fadcc8fe6ae047c1b142514aad8bebba P 311f765f59754bdcb4eadc04e14bd4ea: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:20.688585 19216 tablet_bootstrap.cc:654] T 49ef898a43704b87ba077cd7b0485e9a P da3584f429104d23a1a8899b35cc2138: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:20.693001 19216 tablet_bootstrap.cc:492] T 49ef898a43704b87ba077cd7b0485e9a P da3584f429104d23a1a8899b35cc2138: No bootstrap required, opened a new log
I20240502 01:54:20.693166 19216 ts_tablet_manager.cc:1367] T 49ef898a43704b87ba077cd7b0485e9a P da3584f429104d23a1a8899b35cc2138: Time spent bootstrapping tablet: real 0.010s	user 0.005s	sys 0.000s
I20240502 01:54:20.693774 19213 tablet_bootstrap.cc:492] T df16e178e04040699201f325ce9e2979 P 1b50a031f2784ba498b3e52e16916205: No bootstrap required, opened a new log
I20240502 01:54:20.693840 19216 raft_consensus.cc:348] T 49ef898a43704b87ba077cd7b0485e9a P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } }
I20240502 01:54:20.693989 19213 ts_tablet_manager.cc:1367] T df16e178e04040699201f325ce9e2979 P 1b50a031f2784ba498b3e52e16916205: Time spent bootstrapping tablet: real 0.012s	user 0.001s	sys 0.003s
I20240502 01:54:20.694053 19216 raft_consensus.cc:374] T 49ef898a43704b87ba077cd7b0485e9a P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:20.694151 19216 raft_consensus.cc:729] T 49ef898a43704b87ba077cd7b0485e9a P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da3584f429104d23a1a8899b35cc2138, State: Initialized, Role: FOLLOWER
I20240502 01:54:20.694391 19216 consensus_queue.cc:260] T 49ef898a43704b87ba077cd7b0485e9a P da3584f429104d23a1a8899b35cc2138 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } }
I20240502 01:54:20.694911 19216 ts_tablet_manager.cc:1398] T 49ef898a43704b87ba077cd7b0485e9a P da3584f429104d23a1a8899b35cc2138: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 01:54:20.694967 19213 raft_consensus.cc:348] T df16e178e04040699201f325ce9e2979 P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } }
I20240502 01:54:20.695179 19216 tablet_bootstrap.cc:492] T fadcc8fe6ae047c1b142514aad8bebba P da3584f429104d23a1a8899b35cc2138: Bootstrap starting.
I20240502 01:54:20.695184 19213 raft_consensus.cc:374] T df16e178e04040699201f325ce9e2979 P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:20.695273 19213 raft_consensus.cc:729] T df16e178e04040699201f325ce9e2979 P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b50a031f2784ba498b3e52e16916205, State: Initialized, Role: FOLLOWER
I20240502 01:54:20.695513 19213 consensus_queue.cc:260] T df16e178e04040699201f325ce9e2979 P 1b50a031f2784ba498b3e52e16916205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } }
I20240502 01:54:20.696270 19213 ts_tablet_manager.cc:1398] T df16e178e04040699201f325ce9e2979 P 1b50a031f2784ba498b3e52e16916205: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.000s
I20240502 01:54:20.696676 19213 tablet_bootstrap.cc:492] T 49ef898a43704b87ba077cd7b0485e9a P 1b50a031f2784ba498b3e52e16916205: Bootstrap starting.
I20240502 01:54:20.698300 19216 tablet_bootstrap.cc:654] T fadcc8fe6ae047c1b142514aad8bebba P da3584f429104d23a1a8899b35cc2138: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:20.699234 19213 tablet_bootstrap.cc:654] T 49ef898a43704b87ba077cd7b0485e9a P 1b50a031f2784ba498b3e52e16916205: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:20.700881 19214 tablet_bootstrap.cc:492] T df16e178e04040699201f325ce9e2979 P 84d3a4b174c34e02aedab1e620731359: No bootstrap required, opened a new log
I20240502 01:54:20.701093 19214 ts_tablet_manager.cc:1367] T df16e178e04040699201f325ce9e2979 P 84d3a4b174c34e02aedab1e620731359: Time spent bootstrapping tablet: real 0.020s	user 0.003s	sys 0.004s
I20240502 01:54:20.702252 19214 raft_consensus.cc:348] T df16e178e04040699201f325ce9e2979 P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } }
I20240502 01:54:20.702556 19214 raft_consensus.cc:374] T df16e178e04040699201f325ce9e2979 P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:20.702672 19214 raft_consensus.cc:729] T df16e178e04040699201f325ce9e2979 P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84d3a4b174c34e02aedab1e620731359, State: Initialized, Role: FOLLOWER
I20240502 01:54:20.703123 19214 consensus_queue.cc:260] T df16e178e04040699201f325ce9e2979 P 84d3a4b174c34e02aedab1e620731359 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } }
I20240502 01:54:20.704161 19214 ts_tablet_manager.cc:1398] T df16e178e04040699201f325ce9e2979 P 84d3a4b174c34e02aedab1e620731359: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.001s
I20240502 01:54:20.704607 19214 tablet_bootstrap.cc:492] T fadcc8fe6ae047c1b142514aad8bebba P 84d3a4b174c34e02aedab1e620731359: Bootstrap starting.
I20240502 01:54:20.706990 19214 tablet_bootstrap.cc:654] T fadcc8fe6ae047c1b142514aad8bebba P 84d3a4b174c34e02aedab1e620731359: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:20.709759 19212 tablet_bootstrap.cc:492] T fadcc8fe6ae047c1b142514aad8bebba P 311f765f59754bdcb4eadc04e14bd4ea: No bootstrap required, opened a new log
I20240502 01:54:20.710039 19212 ts_tablet_manager.cc:1367] T fadcc8fe6ae047c1b142514aad8bebba P 311f765f59754bdcb4eadc04e14bd4ea: Time spent bootstrapping tablet: real 0.025s	user 0.002s	sys 0.016s
I20240502 01:54:20.710996 19212 raft_consensus.cc:348] T fadcc8fe6ae047c1b142514aad8bebba P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } }
I20240502 01:54:20.711302 19212 raft_consensus.cc:374] T fadcc8fe6ae047c1b142514aad8bebba P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:20.711669 19212 raft_consensus.cc:729] T fadcc8fe6ae047c1b142514aad8bebba P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 311f765f59754bdcb4eadc04e14bd4ea, State: Initialized, Role: FOLLOWER
I20240502 01:54:20.712113 19212 consensus_queue.cc:260] T fadcc8fe6ae047c1b142514aad8bebba P 311f765f59754bdcb4eadc04e14bd4ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } }
I20240502 01:54:20.714146 19212 ts_tablet_manager.cc:1398] T fadcc8fe6ae047c1b142514aad8bebba P 311f765f59754bdcb4eadc04e14bd4ea: Time spent starting tablet: real 0.004s	user 0.000s	sys 0.002s
I20240502 01:54:20.729326 19216 tablet_bootstrap.cc:492] T fadcc8fe6ae047c1b142514aad8bebba P da3584f429104d23a1a8899b35cc2138: No bootstrap required, opened a new log
I20240502 01:54:20.729640 19216 ts_tablet_manager.cc:1367] T fadcc8fe6ae047c1b142514aad8bebba P da3584f429104d23a1a8899b35cc2138: Time spent bootstrapping tablet: real 0.035s	user 0.003s	sys 0.010s
I20240502 01:54:20.730623 19216 raft_consensus.cc:348] T fadcc8fe6ae047c1b142514aad8bebba P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } }
I20240502 01:54:20.731021 19216 raft_consensus.cc:374] T fadcc8fe6ae047c1b142514aad8bebba P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:20.731155 19216 raft_consensus.cc:729] T fadcc8fe6ae047c1b142514aad8bebba P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da3584f429104d23a1a8899b35cc2138, State: Initialized, Role: FOLLOWER
I20240502 01:54:20.731456 19216 consensus_queue.cc:260] T fadcc8fe6ae047c1b142514aad8bebba P da3584f429104d23a1a8899b35cc2138 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } }
I20240502 01:54:20.732228 19216 ts_tablet_manager.cc:1398] T fadcc8fe6ae047c1b142514aad8bebba P da3584f429104d23a1a8899b35cc2138: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 01:54:20.733695 19214 tablet_bootstrap.cc:492] T fadcc8fe6ae047c1b142514aad8bebba P 84d3a4b174c34e02aedab1e620731359: No bootstrap required, opened a new log
I20240502 01:54:20.733934 19214 ts_tablet_manager.cc:1367] T fadcc8fe6ae047c1b142514aad8bebba P 84d3a4b174c34e02aedab1e620731359: Time spent bootstrapping tablet: real 0.029s	user 0.004s	sys 0.009s
I20240502 01:54:20.734012 19119 raft_consensus.cc:482] T 49ef898a43704b87ba077cd7b0485e9a P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:20.734202 19119 raft_consensus.cc:504] T 49ef898a43704b87ba077cd7b0485e9a P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } }
I20240502 01:54:20.734869 19214 raft_consensus.cc:348] T fadcc8fe6ae047c1b142514aad8bebba P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } }
I20240502 01:54:20.735033 19119 leader_election.cc:288] T 49ef898a43704b87ba077cd7b0485e9a P 311f765f59754bdcb4eadc04e14bd4ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers da3584f429104d23a1a8899b35cc2138 (127.9.176.196:46381), 1b50a031f2784ba498b3e52e16916205 (127.9.176.194:45617)
I20240502 01:54:20.735093 19214 raft_consensus.cc:374] T fadcc8fe6ae047c1b142514aad8bebba P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:20.735175 19214 raft_consensus.cc:729] T fadcc8fe6ae047c1b142514aad8bebba P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84d3a4b174c34e02aedab1e620731359, State: Initialized, Role: FOLLOWER
I20240502 01:54:20.735455 19214 consensus_queue.cc:260] T fadcc8fe6ae047c1b142514aad8bebba P 84d3a4b174c34e02aedab1e620731359 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } }
I20240502 01:54:20.736016 19214 ts_tablet_manager.cc:1398] T fadcc8fe6ae047c1b142514aad8bebba P 84d3a4b174c34e02aedab1e620731359: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240502 01:54:20.736223 17899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49ef898a43704b87ba077cd7b0485e9a" candidate_uuid: "311f765f59754bdcb4eadc04e14bd4ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da3584f429104d23a1a8899b35cc2138" is_pre_election: true
I20240502 01:54:20.736467 17899 raft_consensus.cc:2454] T 49ef898a43704b87ba077cd7b0485e9a P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 311f765f59754bdcb4eadc04e14bd4ea in term 0.
I20240502 01:54:20.736939 17634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49ef898a43704b87ba077cd7b0485e9a" candidate_uuid: "311f765f59754bdcb4eadc04e14bd4ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b50a031f2784ba498b3e52e16916205" is_pre_election: true
I20240502 01:54:20.737382 19213 tablet_bootstrap.cc:492] T 49ef898a43704b87ba077cd7b0485e9a P 1b50a031f2784ba498b3e52e16916205: No bootstrap required, opened a new log
I20240502 01:54:20.737623 19213 ts_tablet_manager.cc:1367] T 49ef898a43704b87ba077cd7b0485e9a P 1b50a031f2784ba498b3e52e16916205: Time spent bootstrapping tablet: real 0.041s	user 0.011s	sys 0.017s
I20240502 01:54:20.738889 19213 raft_consensus.cc:348] T 49ef898a43704b87ba077cd7b0485e9a P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } }
W20240502 01:54:20.739133 17435 leader_election.cc:341] T 49ef898a43704b87ba077cd7b0485e9a P 311f765f59754bdcb4eadc04e14bd4ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1b50a031f2784ba498b3e52e16916205 (127.9.176.194:45617): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:54:20.739593 19213 raft_consensus.cc:374] T 49ef898a43704b87ba077cd7b0485e9a P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:20.739773 19213 raft_consensus.cc:729] T 49ef898a43704b87ba077cd7b0485e9a P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b50a031f2784ba498b3e52e16916205, State: Initialized, Role: FOLLOWER
I20240502 01:54:20.739934 17432 leader_election.cc:302] T 49ef898a43704b87ba077cd7b0485e9a P 311f765f59754bdcb4eadc04e14bd4ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 311f765f59754bdcb4eadc04e14bd4ea, da3584f429104d23a1a8899b35cc2138; no voters: 1b50a031f2784ba498b3e52e16916205
I20240502 01:54:20.740248 19119 raft_consensus.cc:2781] T 49ef898a43704b87ba077cd7b0485e9a P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:20.740375 19119 raft_consensus.cc:482] T 49ef898a43704b87ba077cd7b0485e9a P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:20.740482 19119 raft_consensus.cc:3037] T 49ef898a43704b87ba077cd7b0485e9a P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:20.743297 19119 raft_consensus.cc:504] T 49ef898a43704b87ba077cd7b0485e9a P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } }
I20240502 01:54:20.745399 19119 leader_election.cc:288] T 49ef898a43704b87ba077cd7b0485e9a P 311f765f59754bdcb4eadc04e14bd4ea [CANDIDATE]: Term 1 election: Requested vote from peers da3584f429104d23a1a8899b35cc2138 (127.9.176.196:46381), 1b50a031f2784ba498b3e52e16916205 (127.9.176.194:45617)
I20240502 01:54:20.745824 19213 consensus_queue.cc:260] T 49ef898a43704b87ba077cd7b0485e9a P 1b50a031f2784ba498b3e52e16916205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } }
I20240502 01:54:20.746872 19213 ts_tablet_manager.cc:1398] T 49ef898a43704b87ba077cd7b0485e9a P 1b50a031f2784ba498b3e52e16916205: Time spent starting tablet: real 0.009s	user 0.000s	sys 0.002s
I20240502 01:54:20.747825 17633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49ef898a43704b87ba077cd7b0485e9a" candidate_uuid: "311f765f59754bdcb4eadc04e14bd4ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b50a031f2784ba498b3e52e16916205"
I20240502 01:54:20.748080 17633 raft_consensus.cc:3037] T 49ef898a43704b87ba077cd7b0485e9a P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:20.750594 17633 raft_consensus.cc:2454] T 49ef898a43704b87ba077cd7b0485e9a P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 311f765f59754bdcb4eadc04e14bd4ea in term 1.
I20240502 01:54:20.751662 17435 leader_election.cc:302] T 49ef898a43704b87ba077cd7b0485e9a P 311f765f59754bdcb4eadc04e14bd4ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b50a031f2784ba498b3e52e16916205, 311f765f59754bdcb4eadc04e14bd4ea; no voters: 
I20240502 01:54:20.752180 17899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49ef898a43704b87ba077cd7b0485e9a" candidate_uuid: "311f765f59754bdcb4eadc04e14bd4ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da3584f429104d23a1a8899b35cc2138"
I20240502 01:54:20.752422 17899 raft_consensus.cc:3037] T 49ef898a43704b87ba077cd7b0485e9a P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:20.755009 19119 raft_consensus.cc:2781] T 49ef898a43704b87ba077cd7b0485e9a P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:20.758643 17899 raft_consensus.cc:2454] T 49ef898a43704b87ba077cd7b0485e9a P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 311f765f59754bdcb4eadc04e14bd4ea in term 1.
I20240502 01:54:20.775643 19119 raft_consensus.cc:686] T 49ef898a43704b87ba077cd7b0485e9a P 311f765f59754bdcb4eadc04e14bd4ea [term 1 LEADER]: Becoming Leader. State: Replica: 311f765f59754bdcb4eadc04e14bd4ea, State: Running, Role: LEADER
I20240502 01:54:20.776101 19119 consensus_queue.cc:237] T 49ef898a43704b87ba077cd7b0485e9a P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } }
I20240502 01:54:20.779597 17354 catalog_manager.cc:5484] T 49ef898a43704b87ba077cd7b0485e9a P 311f765f59754bdcb4eadc04e14bd4ea reported cstate change: term changed from 0 to 1, leader changed from <none> to 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193). New cstate: current_term: 1 leader_uuid: "311f765f59754bdcb4eadc04e14bd4ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:20.789737 19145 raft_consensus.cc:482] T df16e178e04040699201f325ce9e2979 P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:20.790016 19145 raft_consensus.cc:504] T df16e178e04040699201f325ce9e2979 P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } }
I20240502 01:54:20.790843 19145 leader_election.cc:288] T df16e178e04040699201f325ce9e2979 P 1b50a031f2784ba498b3e52e16916205 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193:35397), 84d3a4b174c34e02aedab1e620731359 (127.9.176.195:37357)
I20240502 01:54:20.791524 17766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df16e178e04040699201f325ce9e2979" candidate_uuid: "1b50a031f2784ba498b3e52e16916205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84d3a4b174c34e02aedab1e620731359" is_pre_election: true
I20240502 01:54:20.791539 17499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df16e178e04040699201f325ce9e2979" candidate_uuid: "1b50a031f2784ba498b3e52e16916205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "311f765f59754bdcb4eadc04e14bd4ea" is_pre_election: true
I20240502 01:54:20.791800 17766 raft_consensus.cc:2454] T df16e178e04040699201f325ce9e2979 P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b50a031f2784ba498b3e52e16916205 in term 0.
I20240502 01:54:20.791800 17499 raft_consensus.cc:2454] T df16e178e04040699201f325ce9e2979 P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b50a031f2784ba498b3e52e16916205 in term 0.
I20240502 01:54:20.792312 17568 leader_election.cc:302] T df16e178e04040699201f325ce9e2979 P 1b50a031f2784ba498b3e52e16916205 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b50a031f2784ba498b3e52e16916205, 311f765f59754bdcb4eadc04e14bd4ea; no voters: 
I20240502 01:54:20.792644 19145 raft_consensus.cc:2781] T df16e178e04040699201f325ce9e2979 P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:20.792737 19145 raft_consensus.cc:482] T df16e178e04040699201f325ce9e2979 P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:20.792840 19145 raft_consensus.cc:3037] T df16e178e04040699201f325ce9e2979 P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:20.794843 19145 raft_consensus.cc:504] T df16e178e04040699201f325ce9e2979 P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } }
I20240502 01:54:20.795526 19145 leader_election.cc:288] T df16e178e04040699201f325ce9e2979 P 1b50a031f2784ba498b3e52e16916205 [CANDIDATE]: Term 1 election: Requested vote from peers 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193:35397), 84d3a4b174c34e02aedab1e620731359 (127.9.176.195:37357)
I20240502 01:54:20.796092 17499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df16e178e04040699201f325ce9e2979" candidate_uuid: "1b50a031f2784ba498b3e52e16916205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "311f765f59754bdcb4eadc04e14bd4ea"
I20240502 01:54:20.796191 17766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df16e178e04040699201f325ce9e2979" candidate_uuid: "1b50a031f2784ba498b3e52e16916205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84d3a4b174c34e02aedab1e620731359"
I20240502 01:54:20.796289 17499 raft_consensus.cc:3037] T df16e178e04040699201f325ce9e2979 P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:20.796437 17766 raft_consensus.cc:3037] T df16e178e04040699201f325ce9e2979 P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:20.798466 17499 raft_consensus.cc:2454] T df16e178e04040699201f325ce9e2979 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b50a031f2784ba498b3e52e16916205 in term 1.
I20240502 01:54:20.798601 17766 raft_consensus.cc:2454] T df16e178e04040699201f325ce9e2979 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b50a031f2784ba498b3e52e16916205 in term 1.
I20240502 01:54:20.799072 17568 leader_election.cc:302] T df16e178e04040699201f325ce9e2979 P 1b50a031f2784ba498b3e52e16916205 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b50a031f2784ba498b3e52e16916205, 311f765f59754bdcb4eadc04e14bd4ea; no voters: 
I20240502 01:54:20.799427 19145 raft_consensus.cc:2781] T df16e178e04040699201f325ce9e2979 P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:20.799664 19145 raft_consensus.cc:686] T df16e178e04040699201f325ce9e2979 P 1b50a031f2784ba498b3e52e16916205 [term 1 LEADER]: Becoming Leader. State: Replica: 1b50a031f2784ba498b3e52e16916205, State: Running, Role: LEADER
I20240502 01:54:20.799964 19145 consensus_queue.cc:237] T df16e178e04040699201f325ce9e2979 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } }
I20240502 01:54:20.803233 17354 catalog_manager.cc:5484] T df16e178e04040699201f325ce9e2979 P 1b50a031f2784ba498b3e52e16916205 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b50a031f2784ba498b3e52e16916205 (127.9.176.194). New cstate: current_term: 1 leader_uuid: "1b50a031f2784ba498b3e52e16916205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:21.110110 19119 raft_consensus.cc:482] T fadcc8fe6ae047c1b142514aad8bebba P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:21.110347 19119 raft_consensus.cc:504] T fadcc8fe6ae047c1b142514aad8bebba P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } }
I20240502 01:54:21.111016 19119 leader_election.cc:288] T fadcc8fe6ae047c1b142514aad8bebba P 311f765f59754bdcb4eadc04e14bd4ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers da3584f429104d23a1a8899b35cc2138 (127.9.176.196:46381), 84d3a4b174c34e02aedab1e620731359 (127.9.176.195:37357)
I20240502 01:54:21.111620 17899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fadcc8fe6ae047c1b142514aad8bebba" candidate_uuid: "311f765f59754bdcb4eadc04e14bd4ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da3584f429104d23a1a8899b35cc2138" is_pre_election: true
I20240502 01:54:21.111714 17767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fadcc8fe6ae047c1b142514aad8bebba" candidate_uuid: "311f765f59754bdcb4eadc04e14bd4ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84d3a4b174c34e02aedab1e620731359" is_pre_election: true
I20240502 01:54:21.111862 17899 raft_consensus.cc:2454] T fadcc8fe6ae047c1b142514aad8bebba P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 311f765f59754bdcb4eadc04e14bd4ea in term 0.
I20240502 01:54:21.111989 17767 raft_consensus.cc:2454] T fadcc8fe6ae047c1b142514aad8bebba P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 311f765f59754bdcb4eadc04e14bd4ea in term 0.
I20240502 01:54:21.112350 17432 leader_election.cc:302] T fadcc8fe6ae047c1b142514aad8bebba P 311f765f59754bdcb4eadc04e14bd4ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 311f765f59754bdcb4eadc04e14bd4ea, da3584f429104d23a1a8899b35cc2138; no voters: 
I20240502 01:54:21.112710 19119 raft_consensus.cc:2781] T fadcc8fe6ae047c1b142514aad8bebba P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:21.112823 19119 raft_consensus.cc:482] T fadcc8fe6ae047c1b142514aad8bebba P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:21.112951 19119 raft_consensus.cc:3037] T fadcc8fe6ae047c1b142514aad8bebba P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:21.115401 19119 raft_consensus.cc:504] T fadcc8fe6ae047c1b142514aad8bebba P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } }
I20240502 01:54:21.116112 19119 leader_election.cc:288] T fadcc8fe6ae047c1b142514aad8bebba P 311f765f59754bdcb4eadc04e14bd4ea [CANDIDATE]: Term 1 election: Requested vote from peers da3584f429104d23a1a8899b35cc2138 (127.9.176.196:46381), 84d3a4b174c34e02aedab1e620731359 (127.9.176.195:37357)
I20240502 01:54:21.116488 17899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fadcc8fe6ae047c1b142514aad8bebba" candidate_uuid: "311f765f59754bdcb4eadc04e14bd4ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da3584f429104d23a1a8899b35cc2138"
I20240502 01:54:21.116703 17899 raft_consensus.cc:3037] T fadcc8fe6ae047c1b142514aad8bebba P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:21.116626 17767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fadcc8fe6ae047c1b142514aad8bebba" candidate_uuid: "311f765f59754bdcb4eadc04e14bd4ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84d3a4b174c34e02aedab1e620731359"
I20240502 01:54:21.116886 17767 raft_consensus.cc:3037] T fadcc8fe6ae047c1b142514aad8bebba P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:21.118669 17899 raft_consensus.cc:2454] T fadcc8fe6ae047c1b142514aad8bebba P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 311f765f59754bdcb4eadc04e14bd4ea in term 1.
I20240502 01:54:21.118858 17767 raft_consensus.cc:2454] T fadcc8fe6ae047c1b142514aad8bebba P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 311f765f59754bdcb4eadc04e14bd4ea in term 1.
I20240502 01:54:21.119158 17432 leader_election.cc:302] T fadcc8fe6ae047c1b142514aad8bebba P 311f765f59754bdcb4eadc04e14bd4ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 311f765f59754bdcb4eadc04e14bd4ea, da3584f429104d23a1a8899b35cc2138; no voters: 
I20240502 01:54:21.119453 19119 raft_consensus.cc:2781] T fadcc8fe6ae047c1b142514aad8bebba P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:21.119670 19119 raft_consensus.cc:686] T fadcc8fe6ae047c1b142514aad8bebba P 311f765f59754bdcb4eadc04e14bd4ea [term 1 LEADER]: Becoming Leader. State: Replica: 311f765f59754bdcb4eadc04e14bd4ea, State: Running, Role: LEADER
I20240502 01:54:21.119999 19119 consensus_queue.cc:237] T fadcc8fe6ae047c1b142514aad8bebba P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } }
I20240502 01:54:21.124457 17354 catalog_manager.cc:5484] T fadcc8fe6ae047c1b142514aad8bebba P 311f765f59754bdcb4eadc04e14bd4ea reported cstate change: term changed from 0 to 1, leader changed from <none> to 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193). New cstate: current_term: 1 leader_uuid: "311f765f59754bdcb4eadc04e14bd4ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:21.163230 19119 consensus_queue.cc:1035] T 49ef898a43704b87ba077cd7b0485e9a P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:21.167380 19119 consensus_queue.cc:1035] T 49ef898a43704b87ba077cd7b0485e9a P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:21.231914 19145 consensus_queue.cc:1035] T df16e178e04040699201f325ce9e2979 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:21.251516 19202 consensus_queue.cc:1035] T df16e178e04040699201f325ce9e2979 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:21.513723 19225 consensus_queue.cc:1035] T fadcc8fe6ae047c1b142514aad8bebba P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:21.523798 19119 consensus_queue.cc:1035] T fadcc8fe6ae047c1b142514aad8bebba P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:54:21.533078 19229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:54:21.533437 19229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:54:21.542052 19229 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_8'

INSERT report
    rows total: 2000
    time total: 251.879 ms
  time per row: 0.12594 ms
-----------------------------------------------------
Suppressions used:
  count      bytes template
     14       3624 libcrypto.so.1.1
-----------------------------------------------------

I20240502 01:54:22.921591 17347 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56698:
name: "rebalance_test_table_9"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 01:54:22.931556 17747 tablet_service.cc:1467] Processing CreateTablet for tablet 05b67d9912f047bfb6a2bed90b82ae15 (DEFAULT_TABLE table=rebalance_test_table_9 [id=60aa7e7a81564c3a888cc9b68700c9ad]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:22.931969 17879 tablet_service.cc:1467] Processing CreateTablet for tablet 05b67d9912f047bfb6a2bed90b82ae15 (DEFAULT_TABLE table=rebalance_test_table_9 [id=60aa7e7a81564c3a888cc9b68700c9ad]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:22.932127 17747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05b67d9912f047bfb6a2bed90b82ae15. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:22.932121 17880 tablet_service.cc:1467] Processing CreateTablet for tablet 974f67157aa04d3e8b49b264cfa104e8 (DEFAULT_TABLE table=rebalance_test_table_9 [id=60aa7e7a81564c3a888cc9b68700c9ad]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:22.932487 17879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05b67d9912f047bfb6a2bed90b82ae15. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:22.935119 17611 tablet_service.cc:1467] Processing CreateTablet for tablet 05b67d9912f047bfb6a2bed90b82ae15 (DEFAULT_TABLE table=rebalance_test_table_9 [id=60aa7e7a81564c3a888cc9b68700c9ad]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:22.935266 17477 tablet_service.cc:1467] Processing CreateTablet for tablet 0dd8944f9fd542e29072ca501df5e112 (DEFAULT_TABLE table=rebalance_test_table_9 [id=60aa7e7a81564c3a888cc9b68700c9ad]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:22.935645 17746 tablet_service.cc:1467] Processing CreateTablet for tablet 974f67157aa04d3e8b49b264cfa104e8 (DEFAULT_TABLE table=rebalance_test_table_9 [id=60aa7e7a81564c3a888cc9b68700c9ad]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:22.935607 17745 tablet_service.cc:1467] Processing CreateTablet for tablet 0dd8944f9fd542e29072ca501df5e112 (DEFAULT_TABLE table=rebalance_test_table_9 [id=60aa7e7a81564c3a888cc9b68700c9ad]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:22.935742 17477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dd8944f9fd542e29072ca501df5e112. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:22.935741 17611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05b67d9912f047bfb6a2bed90b82ae15. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:22.936074 17746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 974f67157aa04d3e8b49b264cfa104e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:22.936269 17745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dd8944f9fd542e29072ca501df5e112. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:22.939885 17613 tablet_service.cc:1467] Processing CreateTablet for tablet 0dd8944f9fd542e29072ca501df5e112 (DEFAULT_TABLE table=rebalance_test_table_9 [id=60aa7e7a81564c3a888cc9b68700c9ad]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:22.940385 17613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dd8944f9fd542e29072ca501df5e112. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:22.941701 17612 tablet_service.cc:1467] Processing CreateTablet for tablet 974f67157aa04d3e8b49b264cfa104e8 (DEFAULT_TABLE table=rebalance_test_table_9 [id=60aa7e7a81564c3a888cc9b68700c9ad]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:22.942476 17880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 974f67157aa04d3e8b49b264cfa104e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:22.945048 19301 tablet_bootstrap.cc:492] T 05b67d9912f047bfb6a2bed90b82ae15 P 84d3a4b174c34e02aedab1e620731359: Bootstrap starting.
I20240502 01:54:22.945849 19302 tablet_bootstrap.cc:492] T 0dd8944f9fd542e29072ca501df5e112 P 311f765f59754bdcb4eadc04e14bd4ea: Bootstrap starting.
I20240502 01:54:22.947853 19302 tablet_bootstrap.cc:654] T 0dd8944f9fd542e29072ca501df5e112 P 311f765f59754bdcb4eadc04e14bd4ea: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:22.949527 19301 tablet_bootstrap.cc:654] T 05b67d9912f047bfb6a2bed90b82ae15 P 84d3a4b174c34e02aedab1e620731359: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:22.952118 19303 tablet_bootstrap.cc:492] T 05b67d9912f047bfb6a2bed90b82ae15 P da3584f429104d23a1a8899b35cc2138: Bootstrap starting.
I20240502 01:54:22.954471 19303 tablet_bootstrap.cc:654] T 05b67d9912f047bfb6a2bed90b82ae15 P da3584f429104d23a1a8899b35cc2138: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:22.959328 17612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 974f67157aa04d3e8b49b264cfa104e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:22.963620 19303 tablet_bootstrap.cc:492] T 05b67d9912f047bfb6a2bed90b82ae15 P da3584f429104d23a1a8899b35cc2138: No bootstrap required, opened a new log
I20240502 01:54:22.963826 19303 ts_tablet_manager.cc:1367] T 05b67d9912f047bfb6a2bed90b82ae15 P da3584f429104d23a1a8899b35cc2138: Time spent bootstrapping tablet: real 0.012s	user 0.009s	sys 0.001s
I20240502 01:54:22.964805 19303 raft_consensus.cc:348] T 05b67d9912f047bfb6a2bed90b82ae15 P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } }
I20240502 01:54:22.965094 19303 raft_consensus.cc:374] T 05b67d9912f047bfb6a2bed90b82ae15 P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:22.965193 19303 raft_consensus.cc:729] T 05b67d9912f047bfb6a2bed90b82ae15 P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da3584f429104d23a1a8899b35cc2138, State: Initialized, Role: FOLLOWER
I20240502 01:54:22.965446 19303 consensus_queue.cc:260] T 05b67d9912f047bfb6a2bed90b82ae15 P da3584f429104d23a1a8899b35cc2138 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } }
I20240502 01:54:22.966970 19305 tablet_bootstrap.cc:492] T 05b67d9912f047bfb6a2bed90b82ae15 P 1b50a031f2784ba498b3e52e16916205: Bootstrap starting.
I20240502 01:54:22.967068 19303 ts_tablet_manager.cc:1398] T 05b67d9912f047bfb6a2bed90b82ae15 P da3584f429104d23a1a8899b35cc2138: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.002s
I20240502 01:54:22.967334 19303 tablet_bootstrap.cc:492] T 974f67157aa04d3e8b49b264cfa104e8 P da3584f429104d23a1a8899b35cc2138: Bootstrap starting.
I20240502 01:54:22.968866 19305 tablet_bootstrap.cc:654] T 05b67d9912f047bfb6a2bed90b82ae15 P 1b50a031f2784ba498b3e52e16916205: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:22.968905 19303 tablet_bootstrap.cc:654] T 974f67157aa04d3e8b49b264cfa104e8 P da3584f429104d23a1a8899b35cc2138: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:22.970548 19301 tablet_bootstrap.cc:492] T 05b67d9912f047bfb6a2bed90b82ae15 P 84d3a4b174c34e02aedab1e620731359: No bootstrap required, opened a new log
I20240502 01:54:22.970727 19301 ts_tablet_manager.cc:1367] T 05b67d9912f047bfb6a2bed90b82ae15 P 84d3a4b174c34e02aedab1e620731359: Time spent bootstrapping tablet: real 0.026s	user 0.003s	sys 0.003s
I20240502 01:54:22.971612 19301 raft_consensus.cc:348] T 05b67d9912f047bfb6a2bed90b82ae15 P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } }
I20240502 01:54:22.971853 19301 raft_consensus.cc:374] T 05b67d9912f047bfb6a2bed90b82ae15 P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:22.971956 19301 raft_consensus.cc:729] T 05b67d9912f047bfb6a2bed90b82ae15 P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84d3a4b174c34e02aedab1e620731359, State: Initialized, Role: FOLLOWER
I20240502 01:54:22.972214 19301 consensus_queue.cc:260] T 05b67d9912f047bfb6a2bed90b82ae15 P 84d3a4b174c34e02aedab1e620731359 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } }
I20240502 01:54:22.972689 19302 tablet_bootstrap.cc:492] T 0dd8944f9fd542e29072ca501df5e112 P 311f765f59754bdcb4eadc04e14bd4ea: No bootstrap required, opened a new log
I20240502 01:54:22.972870 19302 ts_tablet_manager.cc:1367] T 0dd8944f9fd542e29072ca501df5e112 P 311f765f59754bdcb4eadc04e14bd4ea: Time spent bootstrapping tablet: real 0.027s	user 0.004s	sys 0.013s
I20240502 01:54:22.972980 19301 ts_tablet_manager.cc:1398] T 05b67d9912f047bfb6a2bed90b82ae15 P 84d3a4b174c34e02aedab1e620731359: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.003s
I20240502 01:54:22.973295 19301 tablet_bootstrap.cc:492] T 0dd8944f9fd542e29072ca501df5e112 P 84d3a4b174c34e02aedab1e620731359: Bootstrap starting.
I20240502 01:54:22.973821 19302 raft_consensus.cc:348] T 0dd8944f9fd542e29072ca501df5e112 P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } }
I20240502 01:54:22.974042 19302 raft_consensus.cc:374] T 0dd8944f9fd542e29072ca501df5e112 P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:22.974155 19302 raft_consensus.cc:729] T 0dd8944f9fd542e29072ca501df5e112 P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 311f765f59754bdcb4eadc04e14bd4ea, State: Initialized, Role: FOLLOWER
I20240502 01:54:22.974525 19302 consensus_queue.cc:260] T 0dd8944f9fd542e29072ca501df5e112 P 311f765f59754bdcb4eadc04e14bd4ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } }
I20240502 01:54:22.975200 19301 tablet_bootstrap.cc:654] T 0dd8944f9fd542e29072ca501df5e112 P 84d3a4b174c34e02aedab1e620731359: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:22.976202 19302 ts_tablet_manager.cc:1398] T 0dd8944f9fd542e29072ca501df5e112 P 311f765f59754bdcb4eadc04e14bd4ea: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.002s
I20240502 01:54:22.977795 19301 tablet_bootstrap.cc:492] T 0dd8944f9fd542e29072ca501df5e112 P 84d3a4b174c34e02aedab1e620731359: No bootstrap required, opened a new log
I20240502 01:54:22.977962 19301 ts_tablet_manager.cc:1367] T 0dd8944f9fd542e29072ca501df5e112 P 84d3a4b174c34e02aedab1e620731359: Time spent bootstrapping tablet: real 0.005s	user 0.000s	sys 0.003s
I20240502 01:54:22.978777 19301 raft_consensus.cc:348] T 0dd8944f9fd542e29072ca501df5e112 P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } }
I20240502 01:54:22.978998 19301 raft_consensus.cc:374] T 0dd8944f9fd542e29072ca501df5e112 P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:22.979082 19301 raft_consensus.cc:729] T 0dd8944f9fd542e29072ca501df5e112 P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84d3a4b174c34e02aedab1e620731359, State: Initialized, Role: FOLLOWER
I20240502 01:54:22.979357 19301 consensus_queue.cc:260] T 0dd8944f9fd542e29072ca501df5e112 P 84d3a4b174c34e02aedab1e620731359 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } }
I20240502 01:54:22.980077 19301 ts_tablet_manager.cc:1398] T 0dd8944f9fd542e29072ca501df5e112 P 84d3a4b174c34e02aedab1e620731359: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 01:54:22.980404 19301 tablet_bootstrap.cc:492] T 974f67157aa04d3e8b49b264cfa104e8 P 84d3a4b174c34e02aedab1e620731359: Bootstrap starting.
I20240502 01:54:22.982074 19301 tablet_bootstrap.cc:654] T 974f67157aa04d3e8b49b264cfa104e8 P 84d3a4b174c34e02aedab1e620731359: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:22.984849 19301 tablet_bootstrap.cc:492] T 974f67157aa04d3e8b49b264cfa104e8 P 84d3a4b174c34e02aedab1e620731359: No bootstrap required, opened a new log
I20240502 01:54:22.985004 19301 ts_tablet_manager.cc:1367] T 974f67157aa04d3e8b49b264cfa104e8 P 84d3a4b174c34e02aedab1e620731359: Time spent bootstrapping tablet: real 0.005s	user 0.003s	sys 0.000s
I20240502 01:54:22.985954 19301 raft_consensus.cc:348] T 974f67157aa04d3e8b49b264cfa104e8 P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } }
I20240502 01:54:22.986169 19301 raft_consensus.cc:374] T 974f67157aa04d3e8b49b264cfa104e8 P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:22.986267 19301 raft_consensus.cc:729] T 974f67157aa04d3e8b49b264cfa104e8 P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84d3a4b174c34e02aedab1e620731359, State: Initialized, Role: FOLLOWER
I20240502 01:54:22.986547 19301 consensus_queue.cc:260] T 974f67157aa04d3e8b49b264cfa104e8 P 84d3a4b174c34e02aedab1e620731359 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } }
I20240502 01:54:22.987339 19301 ts_tablet_manager.cc:1398] T 974f67157aa04d3e8b49b264cfa104e8 P 84d3a4b174c34e02aedab1e620731359: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240502 01:54:22.991885 19305 tablet_bootstrap.cc:492] T 05b67d9912f047bfb6a2bed90b82ae15 P 1b50a031f2784ba498b3e52e16916205: No bootstrap required, opened a new log
I20240502 01:54:22.992132 19305 ts_tablet_manager.cc:1367] T 05b67d9912f047bfb6a2bed90b82ae15 P 1b50a031f2784ba498b3e52e16916205: Time spent bootstrapping tablet: real 0.025s	user 0.005s	sys 0.012s
I20240502 01:54:22.992157 19303 tablet_bootstrap.cc:492] T 974f67157aa04d3e8b49b264cfa104e8 P da3584f429104d23a1a8899b35cc2138: No bootstrap required, opened a new log
I20240502 01:54:22.992354 19303 ts_tablet_manager.cc:1367] T 974f67157aa04d3e8b49b264cfa104e8 P da3584f429104d23a1a8899b35cc2138: Time spent bootstrapping tablet: real 0.025s	user 0.003s	sys 0.015s
I20240502 01:54:22.993068 19303 raft_consensus.cc:348] T 974f67157aa04d3e8b49b264cfa104e8 P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } }
I20240502 01:54:22.993052 19305 raft_consensus.cc:348] T 05b67d9912f047bfb6a2bed90b82ae15 P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } }
I20240502 01:54:22.993263 19303 raft_consensus.cc:374] T 974f67157aa04d3e8b49b264cfa104e8 P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:22.993361 19303 raft_consensus.cc:729] T 974f67157aa04d3e8b49b264cfa104e8 P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da3584f429104d23a1a8899b35cc2138, State: Initialized, Role: FOLLOWER
I20240502 01:54:22.993362 19305 raft_consensus.cc:374] T 05b67d9912f047bfb6a2bed90b82ae15 P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:22.993503 19305 raft_consensus.cc:729] T 05b67d9912f047bfb6a2bed90b82ae15 P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b50a031f2784ba498b3e52e16916205, State: Initialized, Role: FOLLOWER
I20240502 01:54:22.993722 19303 consensus_queue.cc:260] T 974f67157aa04d3e8b49b264cfa104e8 P da3584f429104d23a1a8899b35cc2138 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } }
I20240502 01:54:22.993793 19305 consensus_queue.cc:260] T 05b67d9912f047bfb6a2bed90b82ae15 P 1b50a031f2784ba498b3e52e16916205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } }
I20240502 01:54:22.995271 19305 ts_tablet_manager.cc:1398] T 05b67d9912f047bfb6a2bed90b82ae15 P 1b50a031f2784ba498b3e52e16916205: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 01:54:22.995615 19305 tablet_bootstrap.cc:492] T 974f67157aa04d3e8b49b264cfa104e8 P 1b50a031f2784ba498b3e52e16916205: Bootstrap starting.
I20240502 01:54:22.997390 19305 tablet_bootstrap.cc:654] T 974f67157aa04d3e8b49b264cfa104e8 P 1b50a031f2784ba498b3e52e16916205: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:23.005040 19303 ts_tablet_manager.cc:1398] T 974f67157aa04d3e8b49b264cfa104e8 P da3584f429104d23a1a8899b35cc2138: Time spent starting tablet: real 0.013s	user 0.003s	sys 0.009s
I20240502 01:54:23.005831 19305 tablet_bootstrap.cc:492] T 974f67157aa04d3e8b49b264cfa104e8 P 1b50a031f2784ba498b3e52e16916205: No bootstrap required, opened a new log
I20240502 01:54:23.006024 19305 ts_tablet_manager.cc:1367] T 974f67157aa04d3e8b49b264cfa104e8 P 1b50a031f2784ba498b3e52e16916205: Time spent bootstrapping tablet: real 0.010s	user 0.000s	sys 0.009s
I20240502 01:54:23.006701 19305 raft_consensus.cc:348] T 974f67157aa04d3e8b49b264cfa104e8 P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } }
I20240502 01:54:23.006893 19305 raft_consensus.cc:374] T 974f67157aa04d3e8b49b264cfa104e8 P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:23.006969 19305 raft_consensus.cc:729] T 974f67157aa04d3e8b49b264cfa104e8 P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b50a031f2784ba498b3e52e16916205, State: Initialized, Role: FOLLOWER
I20240502 01:54:23.007196 19305 consensus_queue.cc:260] T 974f67157aa04d3e8b49b264cfa104e8 P 1b50a031f2784ba498b3e52e16916205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } }
I20240502 01:54:23.007778 19305 ts_tablet_manager.cc:1398] T 974f67157aa04d3e8b49b264cfa104e8 P 1b50a031f2784ba498b3e52e16916205: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240502 01:54:23.008164 19305 tablet_bootstrap.cc:492] T 0dd8944f9fd542e29072ca501df5e112 P 1b50a031f2784ba498b3e52e16916205: Bootstrap starting.
I20240502 01:54:23.009475 19305 tablet_bootstrap.cc:654] T 0dd8944f9fd542e29072ca501df5e112 P 1b50a031f2784ba498b3e52e16916205: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:23.011200 19305 tablet_bootstrap.cc:492] T 0dd8944f9fd542e29072ca501df5e112 P 1b50a031f2784ba498b3e52e16916205: No bootstrap required, opened a new log
I20240502 01:54:23.011329 19305 ts_tablet_manager.cc:1367] T 0dd8944f9fd542e29072ca501df5e112 P 1b50a031f2784ba498b3e52e16916205: Time spent bootstrapping tablet: real 0.003s	user 0.000s	sys 0.002s
I20240502 01:54:23.012037 19305 raft_consensus.cc:348] T 0dd8944f9fd542e29072ca501df5e112 P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } }
I20240502 01:54:23.012229 19305 raft_consensus.cc:374] T 0dd8944f9fd542e29072ca501df5e112 P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:23.012326 19305 raft_consensus.cc:729] T 0dd8944f9fd542e29072ca501df5e112 P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b50a031f2784ba498b3e52e16916205, State: Initialized, Role: FOLLOWER
I20240502 01:54:23.012524 19305 consensus_queue.cc:260] T 0dd8944f9fd542e29072ca501df5e112 P 1b50a031f2784ba498b3e52e16916205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } }
I20240502 01:54:23.012995 19305 ts_tablet_manager.cc:1398] T 0dd8944f9fd542e29072ca501df5e112 P 1b50a031f2784ba498b3e52e16916205: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240502 01:54:23.031160 19312 raft_consensus.cc:482] T 05b67d9912f047bfb6a2bed90b82ae15 P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:23.031386 19312 raft_consensus.cc:504] T 05b67d9912f047bfb6a2bed90b82ae15 P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } }
I20240502 01:54:23.032224 19312 leader_election.cc:288] T 05b67d9912f047bfb6a2bed90b82ae15 P da3584f429104d23a1a8899b35cc2138 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84d3a4b174c34e02aedab1e620731359 (127.9.176.195:37357), 1b50a031f2784ba498b3e52e16916205 (127.9.176.194:45617)
I20240502 01:54:23.032622 17766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05b67d9912f047bfb6a2bed90b82ae15" candidate_uuid: "da3584f429104d23a1a8899b35cc2138" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84d3a4b174c34e02aedab1e620731359" is_pre_election: true
I20240502 01:54:23.032727 17633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05b67d9912f047bfb6a2bed90b82ae15" candidate_uuid: "da3584f429104d23a1a8899b35cc2138" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b50a031f2784ba498b3e52e16916205" is_pre_election: true
I20240502 01:54:23.032886 17766 raft_consensus.cc:2454] T 05b67d9912f047bfb6a2bed90b82ae15 P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da3584f429104d23a1a8899b35cc2138 in term 0.
I20240502 01:54:23.032984 17633 raft_consensus.cc:2454] T 05b67d9912f047bfb6a2bed90b82ae15 P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da3584f429104d23a1a8899b35cc2138 in term 0.
I20240502 01:54:23.033402 17835 leader_election.cc:302] T 05b67d9912f047bfb6a2bed90b82ae15 P da3584f429104d23a1a8899b35cc2138 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84d3a4b174c34e02aedab1e620731359, da3584f429104d23a1a8899b35cc2138; no voters: 
I20240502 01:54:23.033658 19312 raft_consensus.cc:2781] T 05b67d9912f047bfb6a2bed90b82ae15 P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:23.033802 19312 raft_consensus.cc:482] T 05b67d9912f047bfb6a2bed90b82ae15 P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:23.033902 19312 raft_consensus.cc:3037] T 05b67d9912f047bfb6a2bed90b82ae15 P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:23.036734 19312 raft_consensus.cc:504] T 05b67d9912f047bfb6a2bed90b82ae15 P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } }
I20240502 01:54:23.037267 19312 leader_election.cc:288] T 05b67d9912f047bfb6a2bed90b82ae15 P da3584f429104d23a1a8899b35cc2138 [CANDIDATE]: Term 1 election: Requested vote from peers 84d3a4b174c34e02aedab1e620731359 (127.9.176.195:37357), 1b50a031f2784ba498b3e52e16916205 (127.9.176.194:45617)
I20240502 01:54:23.037665 17766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05b67d9912f047bfb6a2bed90b82ae15" candidate_uuid: "da3584f429104d23a1a8899b35cc2138" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84d3a4b174c34e02aedab1e620731359"
I20240502 01:54:23.037858 17766 raft_consensus.cc:3037] T 05b67d9912f047bfb6a2bed90b82ae15 P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:23.037812 17633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05b67d9912f047bfb6a2bed90b82ae15" candidate_uuid: "da3584f429104d23a1a8899b35cc2138" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b50a031f2784ba498b3e52e16916205"
I20240502 01:54:23.037998 17633 raft_consensus.cc:3037] T 05b67d9912f047bfb6a2bed90b82ae15 P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:23.039738 17766 raft_consensus.cc:2454] T 05b67d9912f047bfb6a2bed90b82ae15 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da3584f429104d23a1a8899b35cc2138 in term 1.
I20240502 01:54:23.039968 17633 raft_consensus.cc:2454] T 05b67d9912f047bfb6a2bed90b82ae15 P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da3584f429104d23a1a8899b35cc2138 in term 1.
I20240502 01:54:23.040134 17835 leader_election.cc:302] T 05b67d9912f047bfb6a2bed90b82ae15 P da3584f429104d23a1a8899b35cc2138 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84d3a4b174c34e02aedab1e620731359, da3584f429104d23a1a8899b35cc2138; no voters: 
I20240502 01:54:23.040459 19312 raft_consensus.cc:2781] T 05b67d9912f047bfb6a2bed90b82ae15 P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:23.040609 19312 raft_consensus.cc:686] T 05b67d9912f047bfb6a2bed90b82ae15 P da3584f429104d23a1a8899b35cc2138 [term 1 LEADER]: Becoming Leader. State: Replica: da3584f429104d23a1a8899b35cc2138, State: Running, Role: LEADER
I20240502 01:54:23.041031 19312 consensus_queue.cc:237] T 05b67d9912f047bfb6a2bed90b82ae15 P da3584f429104d23a1a8899b35cc2138 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } }
I20240502 01:54:23.044052 19145 raft_consensus.cc:482] T 0dd8944f9fd542e29072ca501df5e112 P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:23.044220 19145 raft_consensus.cc:504] T 0dd8944f9fd542e29072ca501df5e112 P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } }
I20240502 01:54:23.044802 19145 leader_election.cc:288] T 0dd8944f9fd542e29072ca501df5e112 P 1b50a031f2784ba498b3e52e16916205 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193:35397), 84d3a4b174c34e02aedab1e620731359 (127.9.176.195:37357)
I20240502 01:54:23.044872 17354 catalog_manager.cc:5484] T 05b67d9912f047bfb6a2bed90b82ae15 P da3584f429104d23a1a8899b35cc2138 reported cstate change: term changed from 0 to 1, leader changed from <none> to da3584f429104d23a1a8899b35cc2138 (127.9.176.196). New cstate: current_term: 1 leader_uuid: "da3584f429104d23a1a8899b35cc2138" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:23.045395 17766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dd8944f9fd542e29072ca501df5e112" candidate_uuid: "1b50a031f2784ba498b3e52e16916205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84d3a4b174c34e02aedab1e620731359" is_pre_election: true
I20240502 01:54:23.045423 17499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dd8944f9fd542e29072ca501df5e112" candidate_uuid: "1b50a031f2784ba498b3e52e16916205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "311f765f59754bdcb4eadc04e14bd4ea" is_pre_election: true
I20240502 01:54:23.045595 17766 raft_consensus.cc:2454] T 0dd8944f9fd542e29072ca501df5e112 P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b50a031f2784ba498b3e52e16916205 in term 0.
I20240502 01:54:23.045686 17499 raft_consensus.cc:2454] T 0dd8944f9fd542e29072ca501df5e112 P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b50a031f2784ba498b3e52e16916205 in term 0.
I20240502 01:54:23.046100 17568 leader_election.cc:302] T 0dd8944f9fd542e29072ca501df5e112 P 1b50a031f2784ba498b3e52e16916205 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b50a031f2784ba498b3e52e16916205, 84d3a4b174c34e02aedab1e620731359; no voters: 
I20240502 01:54:23.046380 19145 raft_consensus.cc:2781] T 0dd8944f9fd542e29072ca501df5e112 P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:23.046495 19145 raft_consensus.cc:482] T 0dd8944f9fd542e29072ca501df5e112 P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:23.046613 19145 raft_consensus.cc:3037] T 0dd8944f9fd542e29072ca501df5e112 P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:23.049109 19145 raft_consensus.cc:504] T 0dd8944f9fd542e29072ca501df5e112 P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } }
I20240502 01:54:23.049650 19145 leader_election.cc:288] T 0dd8944f9fd542e29072ca501df5e112 P 1b50a031f2784ba498b3e52e16916205 [CANDIDATE]: Term 1 election: Requested vote from peers 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193:35397), 84d3a4b174c34e02aedab1e620731359 (127.9.176.195:37357)
I20240502 01:54:23.050067 17499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dd8944f9fd542e29072ca501df5e112" candidate_uuid: "1b50a031f2784ba498b3e52e16916205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "311f765f59754bdcb4eadc04e14bd4ea"
I20240502 01:54:23.050145 17766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dd8944f9fd542e29072ca501df5e112" candidate_uuid: "1b50a031f2784ba498b3e52e16916205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84d3a4b174c34e02aedab1e620731359"
I20240502 01:54:23.050271 17499 raft_consensus.cc:3037] T 0dd8944f9fd542e29072ca501df5e112 P 311f765f59754bdcb4eadc04e14bd4ea [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:23.050352 17766 raft_consensus.cc:3037] T 0dd8944f9fd542e29072ca501df5e112 P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:23.052323 17499 raft_consensus.cc:2454] T 0dd8944f9fd542e29072ca501df5e112 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b50a031f2784ba498b3e52e16916205 in term 1.
I20240502 01:54:23.052353 17766 raft_consensus.cc:2454] T 0dd8944f9fd542e29072ca501df5e112 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b50a031f2784ba498b3e52e16916205 in term 1.
I20240502 01:54:23.052932 17568 leader_election.cc:302] T 0dd8944f9fd542e29072ca501df5e112 P 1b50a031f2784ba498b3e52e16916205 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b50a031f2784ba498b3e52e16916205, 311f765f59754bdcb4eadc04e14bd4ea; no voters: 
I20240502 01:54:23.053236 19145 raft_consensus.cc:2781] T 0dd8944f9fd542e29072ca501df5e112 P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:23.053404 19145 raft_consensus.cc:686] T 0dd8944f9fd542e29072ca501df5e112 P 1b50a031f2784ba498b3e52e16916205 [term 1 LEADER]: Becoming Leader. State: Replica: 1b50a031f2784ba498b3e52e16916205, State: Running, Role: LEADER
I20240502 01:54:23.054008 19145 consensus_queue.cc:237] T 0dd8944f9fd542e29072ca501df5e112 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } }
I20240502 01:54:23.058714 17354 catalog_manager.cc:5484] T 0dd8944f9fd542e29072ca501df5e112 P 1b50a031f2784ba498b3e52e16916205 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b50a031f2784ba498b3e52e16916205 (127.9.176.194). New cstate: current_term: 1 leader_uuid: "1b50a031f2784ba498b3e52e16916205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:23.253556 19145 raft_consensus.cc:482] T 974f67157aa04d3e8b49b264cfa104e8 P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:23.253741 19145 raft_consensus.cc:504] T 974f67157aa04d3e8b49b264cfa104e8 P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } }
I20240502 01:54:23.254421 19145 leader_election.cc:288] T 974f67157aa04d3e8b49b264cfa104e8 P 1b50a031f2784ba498b3e52e16916205 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers da3584f429104d23a1a8899b35cc2138 (127.9.176.196:46381), 84d3a4b174c34e02aedab1e620731359 (127.9.176.195:37357)
I20240502 01:54:23.254895 17900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974f67157aa04d3e8b49b264cfa104e8" candidate_uuid: "1b50a031f2784ba498b3e52e16916205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da3584f429104d23a1a8899b35cc2138" is_pre_election: true
I20240502 01:54:23.255023 17766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974f67157aa04d3e8b49b264cfa104e8" candidate_uuid: "1b50a031f2784ba498b3e52e16916205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84d3a4b174c34e02aedab1e620731359" is_pre_election: true
I20240502 01:54:23.255129 17900 raft_consensus.cc:2454] T 974f67157aa04d3e8b49b264cfa104e8 P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b50a031f2784ba498b3e52e16916205 in term 0.
I20240502 01:54:23.255296 17766 raft_consensus.cc:2454] T 974f67157aa04d3e8b49b264cfa104e8 P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b50a031f2784ba498b3e52e16916205 in term 0.
I20240502 01:54:23.255622 17565 leader_election.cc:302] T 974f67157aa04d3e8b49b264cfa104e8 P 1b50a031f2784ba498b3e52e16916205 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b50a031f2784ba498b3e52e16916205, da3584f429104d23a1a8899b35cc2138; no voters: 
I20240502 01:54:23.255946 19145 raft_consensus.cc:2781] T 974f67157aa04d3e8b49b264cfa104e8 P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:23.256078 19145 raft_consensus.cc:482] T 974f67157aa04d3e8b49b264cfa104e8 P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:23.256165 19145 raft_consensus.cc:3037] T 974f67157aa04d3e8b49b264cfa104e8 P 1b50a031f2784ba498b3e52e16916205 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:23.258092 19145 raft_consensus.cc:504] T 974f67157aa04d3e8b49b264cfa104e8 P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } }
I20240502 01:54:23.258692 19145 leader_election.cc:288] T 974f67157aa04d3e8b49b264cfa104e8 P 1b50a031f2784ba498b3e52e16916205 [CANDIDATE]: Term 1 election: Requested vote from peers da3584f429104d23a1a8899b35cc2138 (127.9.176.196:46381), 84d3a4b174c34e02aedab1e620731359 (127.9.176.195:37357)
I20240502 01:54:23.259176 17900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974f67157aa04d3e8b49b264cfa104e8" candidate_uuid: "1b50a031f2784ba498b3e52e16916205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da3584f429104d23a1a8899b35cc2138"
I20240502 01:54:23.259258 17766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974f67157aa04d3e8b49b264cfa104e8" candidate_uuid: "1b50a031f2784ba498b3e52e16916205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84d3a4b174c34e02aedab1e620731359"
I20240502 01:54:23.259454 17900 raft_consensus.cc:3037] T 974f67157aa04d3e8b49b264cfa104e8 P da3584f429104d23a1a8899b35cc2138 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:23.259474 17766 raft_consensus.cc:3037] T 974f67157aa04d3e8b49b264cfa104e8 P 84d3a4b174c34e02aedab1e620731359 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:23.262430 17766 raft_consensus.cc:2454] T 974f67157aa04d3e8b49b264cfa104e8 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b50a031f2784ba498b3e52e16916205 in term 1.
I20240502 01:54:23.262430 17900 raft_consensus.cc:2454] T 974f67157aa04d3e8b49b264cfa104e8 P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b50a031f2784ba498b3e52e16916205 in term 1.
I20240502 01:54:23.262981 17565 leader_election.cc:302] T 974f67157aa04d3e8b49b264cfa104e8 P 1b50a031f2784ba498b3e52e16916205 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b50a031f2784ba498b3e52e16916205, da3584f429104d23a1a8899b35cc2138; no voters: 
I20240502 01:54:23.263316 19145 raft_consensus.cc:2781] T 974f67157aa04d3e8b49b264cfa104e8 P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:23.263490 19145 raft_consensus.cc:686] T 974f67157aa04d3e8b49b264cfa104e8 P 1b50a031f2784ba498b3e52e16916205 [term 1 LEADER]: Becoming Leader. State: Replica: 1b50a031f2784ba498b3e52e16916205, State: Running, Role: LEADER
I20240502 01:54:23.263847 19145 consensus_queue.cc:237] T 974f67157aa04d3e8b49b264cfa104e8 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } }
I20240502 01:54:23.267256 17354 catalog_manager.cc:5484] T 974f67157aa04d3e8b49b264cfa104e8 P 1b50a031f2784ba498b3e52e16916205 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b50a031f2784ba498b3e52e16916205 (127.9.176.194). New cstate: current_term: 1 leader_uuid: "1b50a031f2784ba498b3e52e16916205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:23.537665 19296 consensus_queue.cc:1035] T 0dd8944f9fd542e29072ca501df5e112 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:23.545169 19321 consensus_queue.cc:1035] T 0dd8944f9fd542e29072ca501df5e112 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:23.619730 19312 consensus_queue.cc:1035] T 05b67d9912f047bfb6a2bed90b82ae15 P da3584f429104d23a1a8899b35cc2138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:23.625706 19327 consensus_queue.cc:1035] T 05b67d9912f047bfb6a2bed90b82ae15 P da3584f429104d23a1a8899b35cc2138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20240502 01:54:23.703819 19319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:54:23.704202 19319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:54:23.711925 19319 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 01:54:23.727170 19321 consensus_queue.cc:1035] T 974f67157aa04d3e8b49b264cfa104e8 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:23.734503 19145 consensus_queue.cc:1035] T 974f67157aa04d3e8b49b264cfa104e8 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_9'

INSERT report
    rows total: 2000
    time total: 311.208 ms
  time per row: 0.155604 ms
I20240502 01:54:24.186880  9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
/tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.197:35011
--local_ip_for_outbound_sockets=127.9.176.197
--tserver_master_addrs=127.9.176.254:46743
--webserver_port=39411
--webserver_interface=127.9.176.197
--builtin_ntp_servers=127.9.176.212:41499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 01:54:24.571271 19382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:54:24.571646 19382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:54:24.571801 19382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:54:24.580121 19382 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:54:24.580271 19382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:54:24.580442 19382 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.197
I20240502 01:54:24.590988 19382 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:41499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.197:35011
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.9.176.197
--webserver_port=39411
--tserver_master_addrs=127.9.176.254:46743
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.197
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:54:24.592523 19382 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:54:24.594422 19382 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:54:24.607791 19389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:24.608259 19388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:24.608820 19391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:54:24.609601 19382 server_base.cc:1034] running on GCE node
I20240502 01:54:24.610247 19382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:54:24.611275 19382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:54:24.612485 19382 hybrid_clock.cc:648] HybridClock initialized: now 1714614864612423 us; error 59 us; skew 500 ppm
I20240502 01:54:24.612818 19382 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:54:24.620832 19382 webserver.cc:460] Webserver started at http://127.9.176.197:39411/ using document root <none> and password file <none>
I20240502 01:54:24.621708 19382 fs_manager.cc:359] Metadata directory not provided
I20240502 01:54:24.621841 19382 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:54:24.627722 19382 fs_manager.cc:709] Time spent opening directory manager: real 0.004s	user 0.003s	sys 0.000s
I20240502 01:54:24.630915 19398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:54:24.632437 19382 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240502 01:54:24.632614 19382 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "29806d3d37354c169484f6db368f8798"
format_stamp: "Formatted at 2024-05-02 01:54:02 on dist-test-slave-j3hh"
I20240502 01:54:24.633499 19382 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:54:24.647325 19382 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:54:24.648356 19382 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:54:24.648674 19382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:54:24.649693 19382 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:54:24.651391 19382 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:54:24.651484 19382 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:54:24.651646 19382 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:54:24.651705 19382 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:54:24.688638 19382 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.197:35011
I20240502 01:54:24.688680 19511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.197:35011 every 8 connection(s)
I20240502 01:54:24.690137 19382 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 01:54:24.694484  9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu as pid 19382
I20240502 01:54:24.695286  9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
/tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.198:36865
--local_ip_for_outbound_sockets=127.9.176.198
--tserver_master_addrs=127.9.176.254:46743
--webserver_port=40127
--webserver_interface=127.9.176.198
--builtin_ntp_servers=127.9.176.212:41499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:54:24.706424 19512 heartbeater.cc:347] Connected to a master server at 127.9.176.254:46743
I20240502 01:54:24.706745 19512 heartbeater.cc:464] Registering TS with master...
I20240502 01:54:24.707641 19512 heartbeater.cc:511] Master 127.9.176.254:46743 requested a full tablet report, sending...
I20240502 01:54:24.709353 17354 ts_manager.cc:194] Re-registered known tserver with Master: 29806d3d37354c169484f6db368f8798 (127.9.176.197:35011)
I20240502 01:54:24.710564 17354 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.176.197:33231
W20240502 01:54:25.096951 19516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:54:25.097421 19516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:54:25.097628 19516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:54:25.110250 19516 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:54:25.110419 19516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:54:25.110584 19516 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.198
I20240502 01:54:25.120416 19516 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:41499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.198:36865
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.9.176.198
--webserver_port=40127
--tserver_master_addrs=127.9.176.254:46743
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.198
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:54:25.121803 19516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:54:25.123828 19516 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:54:25.139375 19521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:25.139953 19522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:25.141103 19524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:54:25.142148 19516 server_base.cc:1034] running on GCE node
I20240502 01:54:25.142858 19516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:54:25.144438 19516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:54:25.145768 19516 hybrid_clock.cc:648] HybridClock initialized: now 1714614865145680 us; error 91 us; skew 500 ppm
I20240502 01:54:25.146163 19516 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:54:25.149040 19516 webserver.cc:460] Webserver started at http://127.9.176.198:40127/ using document root <none> and password file <none>
I20240502 01:54:25.149847 19516 fs_manager.cc:359] Metadata directory not provided
I20240502 01:54:25.150002 19516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:54:25.156222 19516 fs_manager.cc:709] Time spent opening directory manager: real 0.004s	user 0.003s	sys 0.000s
I20240502 01:54:25.159427 19530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:54:25.160969 19516 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240502 01:54:25.161165 19516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "4b7c382c9edd41babdd1a1afc904a813"
format_stamp: "Formatted at 2024-05-02 01:54:03 on dist-test-slave-j3hh"
I20240502 01:54:25.162052 19516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:54:25.208067 19516 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:54:25.209188 19516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:54:25.209563 19516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:54:25.212873 19516 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:54:25.215332 19516 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:54:25.215472 19516 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.001s	sys 0.000s
I20240502 01:54:25.215641 19516 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:54:25.215724 19516 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:54:25.304358 19516 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.198:36865
I20240502 01:54:25.304458 19649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.198:36865 every 8 connection(s)
I20240502 01:54:25.306094 19516 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 01:54:25.311424  9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu as pid 19516
I20240502 01:54:25.312247  9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
/tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.199:42269
--local_ip_for_outbound_sockets=127.9.176.199
--tserver_master_addrs=127.9.176.254:46743
--webserver_port=45347
--webserver_interface=127.9.176.199
--builtin_ntp_servers=127.9.176.212:41499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:54:25.322423 19650 heartbeater.cc:347] Connected to a master server at 127.9.176.254:46743
I20240502 01:54:25.322814 19650 heartbeater.cc:464] Registering TS with master...
I20240502 01:54:25.323907 19650 heartbeater.cc:511] Master 127.9.176.254:46743 requested a full tablet report, sending...
I20240502 01:54:25.325699 17354 ts_manager.cc:194] Re-registered known tserver with Master: 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198:36865)
I20240502 01:54:25.326623 17354 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.176.198:58917
W20240502 01:54:25.690433 19654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:54:25.690742 19654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:54:25.690876 19654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:54:25.699805 19654 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:54:25.699985 19654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:54:25.700155 19654 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.199
I20240502 01:54:25.713937 19512 heartbeater.cc:503] Master 127.9.176.254:46743 was elected leader, sending a full tablet report...
I20240502 01:54:25.715668 19654 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:41499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.199:42269
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.9.176.199
--webserver_port=45347
--tserver_master_addrs=127.9.176.254:46743
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.199
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:54:25.717449 19654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:54:25.719951 19654 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:54:25.736801 19659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:25.737423 19660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:25.738055 19662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:54:25.738852 19654 server_base.cc:1034] running on GCE node
I20240502 01:54:25.739522 19654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:54:25.740656 19654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:54:25.741894 19654 hybrid_clock.cc:648] HybridClock initialized: now 1714614865741847 us; error 42 us; skew 500 ppm
I20240502 01:54:25.742251 19654 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:54:25.744961 19654 webserver.cc:460] Webserver started at http://127.9.176.199:45347/ using document root <none> and password file <none>
I20240502 01:54:25.745855 19654 fs_manager.cc:359] Metadata directory not provided
I20240502 01:54:25.745985 19654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:54:25.752362 19654 fs_manager.cc:709] Time spent opening directory manager: real 0.004s	user 0.003s	sys 0.000s
I20240502 01:54:25.755784 19668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:54:25.757453 19654 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20240502 01:54:25.757689 19654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "272b2179c1b4476090a89880df2a8baa"
format_stamp: "Formatted at 2024-05-02 01:54:03 on dist-test-slave-j3hh"
I20240502 01:54:25.758639 19654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:54:25.783372 19654 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:54:25.784368 19654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:54:25.784680 19654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:54:25.785804 19654 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:54:25.787508 19654 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:54:25.787622 19654 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.001s
I20240502 01:54:25.787760 19654 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:54:25.787815 19654 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:54:25.823554 19654 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.199:42269
I20240502 01:54:25.823632 19782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.199:42269 every 8 connection(s)
I20240502 01:54:25.825260 19654 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 01:54:25.826082  9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu as pid 19654
I20240502 01:54:25.827049  9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
/tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.200:37831
--local_ip_for_outbound_sockets=127.9.176.200
--tserver_master_addrs=127.9.176.254:46743
--webserver_port=39509
--webserver_interface=127.9.176.200
--builtin_ntp_servers=127.9.176.212:41499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:54:25.843536 19783 heartbeater.cc:347] Connected to a master server at 127.9.176.254:46743
I20240502 01:54:25.843930 19783 heartbeater.cc:464] Registering TS with master...
I20240502 01:54:25.844918 19783 heartbeater.cc:511] Master 127.9.176.254:46743 requested a full tablet report, sending...
I20240502 01:54:25.846685 17354 ts_manager.cc:194] Re-registered known tserver with Master: 272b2179c1b4476090a89880df2a8baa (127.9.176.199:42269)
I20240502 01:54:25.847783 17354 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.176.199:37703
W20240502 01:54:26.200731 19787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:54:26.201016 19787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:54:26.201124 19787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:54:26.208943 19787 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:54:26.209093 19787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:54:26.209280 19787 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.200
I20240502 01:54:26.223327 19787 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:41499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.200:37831
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.9.176.200
--webserver_port=39509
--tserver_master_addrs=127.9.176.254:46743
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.200
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:54:26.224958 19787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:54:26.227452 19787 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:54:26.240260 19793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:26.240665 19794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:26.241238 19796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:54:26.241920 19787 server_base.cc:1034] running on GCE node
I20240502 01:54:26.242497 19787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:54:26.243393 19787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:54:26.244627 19787 hybrid_clock.cc:648] HybridClock initialized: now 1714614866244562 us; error 52 us; skew 500 ppm
I20240502 01:54:26.244968 19787 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:54:26.247390 19787 webserver.cc:460] Webserver started at http://127.9.176.200:39509/ using document root <none> and password file <none>
I20240502 01:54:26.248231 19787 fs_manager.cc:359] Metadata directory not provided
I20240502 01:54:26.248363 19787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:54:26.254129 19787 fs_manager.cc:709] Time spent opening directory manager: real 0.003s	user 0.005s	sys 0.000s
I20240502 01:54:26.257107 19802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:54:26.258443 19787 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.000s	sys 0.003s
I20240502 01:54:26.258625 19787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "94931f4b51c9493cb3d4264ce99f5e76"
format_stamp: "Formatted at 2024-05-02 01:54:04 on dist-test-slave-j3hh"
I20240502 01:54:26.259438 19787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:54:26.304909 19787 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:54:26.305744 19787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:54:26.306016 19787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:54:26.307034 19787 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:54:26.308854 19787 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:54:26.308933 19787 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:54:26.309055 19787 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:54:26.309108 19787 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:54:26.329712 19650 heartbeater.cc:503] Master 127.9.176.254:46743 was elected leader, sending a full tablet report...
I20240502 01:54:26.344668 19787 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.200:37831
I20240502 01:54:26.344709 19916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.200:37831 every 8 connection(s)
I20240502 01:54:26.346313 19787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 01:54:26.347954  9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu as pid 19787
I20240502 01:54:26.363137 19917 heartbeater.cc:347] Connected to a master server at 127.9.176.254:46743
I20240502 01:54:26.363646 19917 heartbeater.cc:464] Registering TS with master...
I20240502 01:54:26.365202 19917 heartbeater.cc:511] Master 127.9.176.254:46743 requested a full tablet report, sending...
I20240502 01:54:26.367308 17354 ts_manager.cc:194] Re-registered known tserver with Master: 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200:37831)
I20240502 01:54:26.368402 17354 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.176.200:46271
I20240502 01:54:26.852686 19783 heartbeater.cc:503] Master 127.9.176.254:46743 was elected leader, sending a full tablet report...
I20240502 01:54:27.376579 19917 heartbeater.cc:503] Master 127.9.176.254:46743 was elected leader, sending a full tablet report...
I20240502 01:54:27.871371 17477 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:54:27.883544 19446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:27.901063 19585 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:27.956219 17746 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:54:27.983100 17613 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:54:28.025242 17880 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:54:28.044502 19446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:54:28.043871 17435 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.176.194:45617, user_credentials={real_user=slave}} blocked reactor thread for 102623us
I20240502 01:54:28.045863 19718 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:28.052852 19852 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:28.057602 17746 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:54:28.060243 19852 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:28.066815 17477 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:54:28.088642 17613 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:54:28.106490 17880 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:54:28.113684 17477 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:54:28.124300 19446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:28.129475 19718 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:28.131049 17613 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:54:28.145104 19718 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:28.173666 19585 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:28.190248 19585 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:28.223268 17746 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:54:28.266132 17880 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:54:28.267148 19852 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:28.886823 17613 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:54:28.893962 19446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:28.939203 19852 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:28.954591 17477 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:54:28.970027 19718 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:28.970721 19585 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:28.986397 17880 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:54:29.003608 17746 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:54:29.078380 17613 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:54:29.099556 19718 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:29.151804 19446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:29.181005 19585 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:29.216161 17477 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:54:29.228067 17746 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:54:29.310050 19852 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:29.353585 17880 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:54:29.439546 19718 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:29.506819 17613 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:54:29.517104 19585 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:29.526142 17477 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:54:29.550937 19446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:29.563970 17746 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:54:29.650496 19852 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:29.652359 17880 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:54:29.659919 19718 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:29.685853 17613 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:54:29.696727 19446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:29.700781 17477 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:54:29.745429 17746 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:54:29.754078 19585 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:29.780465 19852 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:29.798106 17880 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:54:30.099963 17613 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:54:30.108661 19718 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:30.159255 19446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:30.171763 17477 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:54:30.216490 19585 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:30.220522 17746 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:54:30.297508 19852 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:30.312847 17880 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:54:30.597568 17568 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.176.195:37357, user_credentials={real_user=slave}} blocked reactor thread for 47792.1us
I20240502 01:54:30.796207 17613 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:54:30.829118 19446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:30.848888 17477 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:54:30.857473 19718 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:30.897102 19852 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:30.905313 19585 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:30.945746 17746 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:54:30.993063 17880 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:54:31.126838 19446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:31.137795 17613 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:54:31.182965 19718 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:31.204888 17880 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:54:31.206655 19718 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:31.209825 17613 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:54:31.238173 19852 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:31.280225 19718 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:31.285059 17477 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:54:31.288067 17613 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:54:31.298772 19585 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:31.316032 19446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:31.329506 17880 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:54:31.331943 17477 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:54:31.339712 17746 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:54:31.362882 17746 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:54:31.408133 19446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:31.434962 19585 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:31.452862 17746 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:54:31.452970 17477 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:54:31.480866 19852 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:31.481729 17880 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:54:31.490418 19585 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:31.522018 19852 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:32.010813 19718 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:32.012512 19585 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:32.044056 19446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:32.053134 17634 consensus_queue.cc:237] T 3b3fa060e8d64911bec810ec07a358d2 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } }
I20240502 01:54:32.062733 17746 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:54:32.063489 19852 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:32.092032 17477 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:54:32.094695 17880 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:54:32.144526 17700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.176.196:46381, user_credentials={real_user=slave}} blocked reactor thread for 74670.4us
I20240502 01:54:32.154006 17633 consensus_queue.cc:237] T 91e0e82526c0489a8908e250bf3c851f P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } }
I20240502 01:54:32.185560 17497 raft_consensus.cc:1261] T 3b3fa060e8d64911bec810ec07a358d2 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:32.186568 19788 consensus_queue.cc:1035] T 3b3fa060e8d64911bec810ec07a358d2 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:32.201701 17499 raft_consensus.cc:1261] T 91e0e82526c0489a8908e250bf3c851f P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:32.202498 19788 consensus_queue.cc:1035] T 91e0e82526c0489a8908e250bf3c851f P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:32.203965 17613 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 01:54:32.241612 17835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.176.195:37357, user_credentials={real_user=slave}} blocked reactor thread for 82827us
I20240502 01:54:32.244557 19788 raft_consensus.cc:2932] T 3b3fa060e8d64911bec810ec07a358d2 P 1b50a031f2784ba498b3e52e16916205 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 272b2179c1b4476090a89880df2a8baa (127.9.176.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } } }
I20240502 01:54:32.249006 17499 raft_consensus.cc:2932] T 91e0e82526c0489a8908e250bf3c851f P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 29806d3d37354c169484f6db368f8798 (127.9.176.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } } }
I20240502 01:54:32.255669 20628 raft_consensus.cc:2932] T 91e0e82526c0489a8908e250bf3c851f P 1b50a031f2784ba498b3e52e16916205 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 29806d3d37354c169484f6db368f8798 (127.9.176.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } } }
I20240502 01:54:32.259152 17350 catalog_manager.cc:5484] T 3b3fa060e8d64911bec810ec07a358d2 P 1b50a031f2784ba498b3e52e16916205 reported cstate change: config changed from index -1 to 4, NON_VOTER 272b2179c1b4476090a89880df2a8baa (127.9.176.199) added. New cstate: current_term: 1 leader_uuid: "1b50a031f2784ba498b3e52e16916205" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 01:54:32.280442 17565 consensus_peers.cc:485] T 91e0e82526c0489a8908e250bf3c851f P 1b50a031f2784ba498b3e52e16916205 -> Peer 29806d3d37354c169484f6db368f8798 (127.9.176.197:35011): Couldn't send request to peer 29806d3d37354c169484f6db368f8798. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 91e0e82526c0489a8908e250bf3c851f. This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:32.282035 17565 consensus_peers.cc:485] T 3b3fa060e8d64911bec810ec07a358d2 P 1b50a031f2784ba498b3e52e16916205 -> Peer 272b2179c1b4476090a89880df2a8baa (127.9.176.199:42269): Couldn't send request to peer 272b2179c1b4476090a89880df2a8baa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3b3fa060e8d64911bec810ec07a358d2. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:32.287868 17900 raft_consensus.cc:1261] T 91e0e82526c0489a8908e250bf3c851f P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:32.288839 17900 raft_consensus.cc:1261] T 3b3fa060e8d64911bec810ec07a358d2 P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:32.291167 19788 consensus_queue.cc:1035] T 91e0e82526c0489a8908e250bf3c851f P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:32.292646 19788 consensus_queue.cc:1035] T 3b3fa060e8d64911bec810ec07a358d2 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:32.300973 17900 raft_consensus.cc:2932] T 91e0e82526c0489a8908e250bf3c851f P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 29806d3d37354c169484f6db368f8798 (127.9.176.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } } }
I20240502 01:54:32.335467 17499 raft_consensus.cc:2932] T 3b3fa060e8d64911bec810ec07a358d2 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 272b2179c1b4476090a89880df2a8baa (127.9.176.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } } }
I20240502 01:54:32.325618 17348 catalog_manager.cc:5484] T 91e0e82526c0489a8908e250bf3c851f P 311f765f59754bdcb4eadc04e14bd4ea reported cstate change: config changed from index -1 to 4, NON_VOTER 29806d3d37354c169484f6db368f8798 (127.9.176.197) added. New cstate: current_term: 1 leader_uuid: "1b50a031f2784ba498b3e52e16916205" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } } }
I20240502 01:54:32.366994 17899 raft_consensus.cc:2932] T 3b3fa060e8d64911bec810ec07a358d2 P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 272b2179c1b4476090a89880df2a8baa (127.9.176.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } } }
I20240502 01:54:32.405134 17766 consensus_queue.cc:237] T 6da59c0626c64a5988e39a831c42c268 P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } }
I20240502 01:54:32.421420 17499 raft_consensus.cc:1261] T 6da59c0626c64a5988e39a831c42c268 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:32.422914 20293 consensus_queue.cc:1035] T 6da59c0626c64a5988e39a831c42c268 P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:32.429602 17899 raft_consensus.cc:1261] T 6da59c0626c64a5988e39a831c42c268 P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:32.430995 20293 consensus_queue.cc:1035] T 6da59c0626c64a5988e39a831c42c268 P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 01:54:32.432022 17700 consensus_peers.cc:485] T 6da59c0626c64a5988e39a831c42c268 P 84d3a4b174c34e02aedab1e620731359 -> Peer 29806d3d37354c169484f6db368f8798 (127.9.176.197:35011): Couldn't send request to peer 29806d3d37354c169484f6db368f8798. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6da59c0626c64a5988e39a831c42c268. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:32.445960 20293 raft_consensus.cc:2932] T 6da59c0626c64a5988e39a831c42c268 P 84d3a4b174c34e02aedab1e620731359 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 29806d3d37354c169484f6db368f8798 (127.9.176.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } } }
I20240502 01:54:32.453179 17900 raft_consensus.cc:2932] T 6da59c0626c64a5988e39a831c42c268 P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 29806d3d37354c169484f6db368f8798 (127.9.176.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } } }
I20240502 01:54:32.460616 17499 raft_consensus.cc:2932] T 6da59c0626c64a5988e39a831c42c268 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 29806d3d37354c169484f6db368f8798 (127.9.176.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } } }
I20240502 01:54:32.466437 17355 catalog_manager.cc:5484] T 6da59c0626c64a5988e39a831c42c268 P da3584f429104d23a1a8899b35cc2138 reported cstate change: config changed from index -1 to 4, NON_VOTER 29806d3d37354c169484f6db368f8798 (127.9.176.197) added. New cstate: current_term: 1 leader_uuid: "84d3a4b174c34e02aedab1e620731359" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } } }
I20240502 01:54:32.490166 17765 consensus_queue.cc:237] T 768e7cbf63a94d3a815ac77a4d0acf41 P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } }
I20240502 01:54:32.520206 17634 raft_consensus.cc:1261] T 768e7cbf63a94d3a815ac77a4d0acf41 P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:32.521464 17899 raft_consensus.cc:1261] T 768e7cbf63a94d3a815ac77a4d0acf41 P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:32.522293 20618 consensus_queue.cc:1035] T 768e7cbf63a94d3a815ac77a4d0acf41 P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:32.523190 20618 consensus_queue.cc:1035] T 768e7cbf63a94d3a815ac77a4d0acf41 P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 01:54:32.540620 17700 consensus_peers.cc:485] T 768e7cbf63a94d3a815ac77a4d0acf41 P 84d3a4b174c34e02aedab1e620731359 -> Peer 272b2179c1b4476090a89880df2a8baa (127.9.176.199:42269): Couldn't send request to peer 272b2179c1b4476090a89880df2a8baa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 768e7cbf63a94d3a815ac77a4d0acf41. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:32.573168 20618 raft_consensus.cc:2932] T 768e7cbf63a94d3a815ac77a4d0acf41 P 84d3a4b174c34e02aedab1e620731359 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 272b2179c1b4476090a89880df2a8baa (127.9.176.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } } }
I20240502 01:54:32.576694 17900 raft_consensus.cc:2932] T 768e7cbf63a94d3a815ac77a4d0acf41 P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 272b2179c1b4476090a89880df2a8baa (127.9.176.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } } }
I20240502 01:54:32.590824 17349 catalog_manager.cc:5484] T 768e7cbf63a94d3a815ac77a4d0acf41 P da3584f429104d23a1a8899b35cc2138 reported cstate change: config changed from index -1 to 4, NON_VOTER 272b2179c1b4476090a89880df2a8baa (127.9.176.199) added. New cstate: current_term: 1 leader_uuid: "84d3a4b174c34e02aedab1e620731359" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } } }
I20240502 01:54:32.597715 17634 raft_consensus.cc:2932] T 768e7cbf63a94d3a815ac77a4d0acf41 P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 272b2179c1b4476090a89880df2a8baa (127.9.176.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } } }
I20240502 01:54:32.664373 17899 consensus_queue.cc:237] T 3bdd49939e994240a6d15dbd11559ff4 P da3584f429104d23a1a8899b35cc2138 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } }
I20240502 01:54:32.669819 17634 raft_consensus.cc:1261] T 3bdd49939e994240a6d15dbd11559ff4 P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Refusing update from remote peer da3584f429104d23a1a8899b35cc2138: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:32.670102 17765 raft_consensus.cc:1261] T 3bdd49939e994240a6d15dbd11559ff4 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Refusing update from remote peer da3584f429104d23a1a8899b35cc2138: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:32.670936 20321 consensus_queue.cc:1035] T 3bdd49939e994240a6d15dbd11559ff4 P da3584f429104d23a1a8899b35cc2138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:32.671861 20623 consensus_queue.cc:1035] T 3bdd49939e994240a6d15dbd11559ff4 P da3584f429104d23a1a8899b35cc2138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:32.703009 20699 ts_tablet_manager.cc:902] T 3b3fa060e8d64911bec810ec07a358d2 P 272b2179c1b4476090a89880df2a8baa: Initiating tablet copy from peer 1b50a031f2784ba498b3e52e16916205 (127.9.176.194:45617)
I20240502 01:54:32.713075 20623 raft_consensus.cc:2932] T 3bdd49939e994240a6d15dbd11559ff4 P da3584f429104d23a1a8899b35cc2138 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } } }
I20240502 01:54:32.716601 17634 raft_consensus.cc:2932] T 3bdd49939e994240a6d15dbd11559ff4 P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } } }
I20240502 01:54:32.721232 17766 raft_consensus.cc:2932] T 3bdd49939e994240a6d15dbd11559ff4 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } } }
I20240502 01:54:32.723346 17355 catalog_manager.cc:5484] T 3bdd49939e994240a6d15dbd11559ff4 P da3584f429104d23a1a8899b35cc2138 reported cstate change: config changed from index -1 to 4, NON_VOTER 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) added. New cstate: current_term: 1 leader_uuid: "da3584f429104d23a1a8899b35cc2138" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 01:54:32.767684 17833 consensus_peers.cc:485] T 3bdd49939e994240a6d15dbd11559ff4 P da3584f429104d23a1a8899b35cc2138 -> Peer 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200:37831): Couldn't send request to peer 94931f4b51c9493cb3d4264ce99f5e76. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3bdd49939e994240a6d15dbd11559ff4. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:32.781363 17899 consensus_queue.cc:237] T 05b67d9912f047bfb6a2bed90b82ae15 P da3584f429104d23a1a8899b35cc2138 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } }
I20240502 01:54:32.784904 20708 ts_tablet_manager.cc:902] T 91e0e82526c0489a8908e250bf3c851f P 29806d3d37354c169484f6db368f8798: Initiating tablet copy from peer 1b50a031f2784ba498b3e52e16916205 (127.9.176.194:45617)
I20240502 01:54:32.796118 17499 consensus_queue.cc:237] T 37eda67598fe491bbb29aa62b2c4bec7 P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } }
I20240502 01:54:32.797354 17766 raft_consensus.cc:1261] T 05b67d9912f047bfb6a2bed90b82ae15 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Refusing update from remote peer da3584f429104d23a1a8899b35cc2138: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:32.798048 17632 raft_consensus.cc:1261] T 05b67d9912f047bfb6a2bed90b82ae15 P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Refusing update from remote peer da3584f429104d23a1a8899b35cc2138: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:32.798841 20625 consensus_queue.cc:1035] T 05b67d9912f047bfb6a2bed90b82ae15 P da3584f429104d23a1a8899b35cc2138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:32.800155 20623 consensus_queue.cc:1035] T 05b67d9912f047bfb6a2bed90b82ae15 P da3584f429104d23a1a8899b35cc2138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:32.847438 17634 raft_consensus.cc:1261] T 37eda67598fe491bbb29aa62b2c4bec7 P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Refusing update from remote peer 311f765f59754bdcb4eadc04e14bd4ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:32.848342 20585 consensus_queue.cc:1035] T 37eda67598fe491bbb29aa62b2c4bec7 P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:32.848201 20625 raft_consensus.cc:2932] T 05b67d9912f047bfb6a2bed90b82ae15 P da3584f429104d23a1a8899b35cc2138 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } } }
I20240502 01:54:32.856575 17634 raft_consensus.cc:2932] T 05b67d9912f047bfb6a2bed90b82ae15 P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } } }
I20240502 01:54:32.858048 17355 catalog_manager.cc:5484] T 05b67d9912f047bfb6a2bed90b82ae15 P da3584f429104d23a1a8899b35cc2138 reported cstate change: config changed from index -1 to 4, NON_VOTER 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) added. New cstate: current_term: 1 leader_uuid: "da3584f429104d23a1a8899b35cc2138" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:32.865010 17765 raft_consensus.cc:2932] T 05b67d9912f047bfb6a2bed90b82ae15 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } } }
W20240502 01:54:32.881364 17832 consensus_peers.cc:485] T 05b67d9912f047bfb6a2bed90b82ae15 P da3584f429104d23a1a8899b35cc2138 -> Peer 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198:36865): Couldn't send request to peer 4b7c382c9edd41babdd1a1afc904a813. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 05b67d9912f047bfb6a2bed90b82ae15. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:32.888864 17765 raft_consensus.cc:1261] T 37eda67598fe491bbb29aa62b2c4bec7 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Refusing update from remote peer 311f765f59754bdcb4eadc04e14bd4ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:32.889693 20585 consensus_queue.cc:1035] T 37eda67598fe491bbb29aa62b2c4bec7 P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:32.908301 17766 consensus_queue.cc:237] T c91f74f46b04425aa5393515cb9f781c P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } }
I20240502 01:54:32.914736 17899 raft_consensus.cc:1261] T c91f74f46b04425aa5393515cb9f781c P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:32.915601 20618 consensus_queue.cc:1035] T c91f74f46b04425aa5393515cb9f781c P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 01:54:32.916525 17700 consensus_peers.cc:485] T c91f74f46b04425aa5393515cb9f781c P 84d3a4b174c34e02aedab1e620731359 -> Peer 29806d3d37354c169484f6db368f8798 (127.9.176.197:35011): Couldn't send request to peer 29806d3d37354c169484f6db368f8798. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c91f74f46b04425aa5393515cb9f781c. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:32.922953 17499 raft_consensus.cc:1261] T c91f74f46b04425aa5393515cb9f781c P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:32.925079 20618 consensus_queue.cc:1035] T c91f74f46b04425aa5393515cb9f781c P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:32.933364 20732 ts_tablet_manager.cc:902] T 6da59c0626c64a5988e39a831c42c268 P 29806d3d37354c169484f6db368f8798: Initiating tablet copy from peer 84d3a4b174c34e02aedab1e620731359 (127.9.176.195:37357)
I20240502 01:54:32.937474 20585 raft_consensus.cc:2932] T 37eda67598fe491bbb29aa62b2c4bec7 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } } }
I20240502 01:54:32.951186 17350 catalog_manager.cc:5484] T 37eda67598fe491bbb29aa62b2c4bec7 P 311f765f59754bdcb4eadc04e14bd4ea reported cstate change: config changed from index -1 to 4, NON_VOTER 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) added. New cstate: current_term: 1 leader_uuid: "311f765f59754bdcb4eadc04e14bd4ea" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:32.956991 17766 raft_consensus.cc:2932] T 37eda67598fe491bbb29aa62b2c4bec7 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } } }
W20240502 01:54:32.977406 17432 consensus_peers.cc:485] T 37eda67598fe491bbb29aa62b2c4bec7 P 311f765f59754bdcb4eadc04e14bd4ea -> Peer 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198:36865): Couldn't send request to peer 4b7c382c9edd41babdd1a1afc904a813. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 37eda67598fe491bbb29aa62b2c4bec7. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:32.982317 20699 tablet_copy_client.cc:308] T 3b3fa060e8d64911bec810ec07a358d2 P 272b2179c1b4476090a89880df2a8baa: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.194:45617
I20240502 01:54:32.989635 20618 raft_consensus.cc:2932] T c91f74f46b04425aa5393515cb9f781c P 84d3a4b174c34e02aedab1e620731359 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 29806d3d37354c169484f6db368f8798 (127.9.176.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } } }
I20240502 01:54:32.992906 17900 raft_consensus.cc:2932] T c91f74f46b04425aa5393515cb9f781c P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 29806d3d37354c169484f6db368f8798 (127.9.176.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } } }
I20240502 01:54:33.001060 20744 ts_tablet_manager.cc:902] T 768e7cbf63a94d3a815ac77a4d0acf41 P 272b2179c1b4476090a89880df2a8baa: Initiating tablet copy from peer 84d3a4b174c34e02aedab1e620731359 (127.9.176.195:37357)
I20240502 01:54:33.003785 17355 catalog_manager.cc:5484] T c91f74f46b04425aa5393515cb9f781c P 84d3a4b174c34e02aedab1e620731359 reported cstate change: config changed from index -1 to 4, NON_VOTER 29806d3d37354c169484f6db368f8798 (127.9.176.197) added. New cstate: current_term: 1 leader_uuid: "84d3a4b174c34e02aedab1e620731359" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:33.008136 17498 raft_consensus.cc:2932] T c91f74f46b04425aa5393515cb9f781c P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 29806d3d37354c169484f6db368f8798 (127.9.176.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } } }
I20240502 01:54:33.022287 17654 tablet_copy_service.cc:140] P 1b50a031f2784ba498b3e52e16916205: Received BeginTabletCopySession request for tablet 3b3fa060e8d64911bec810ec07a358d2 from peer 272b2179c1b4476090a89880df2a8baa ({username='slave'} at 127.9.176.199:56005)
I20240502 01:54:33.022641 17654 tablet_copy_service.cc:161] P 1b50a031f2784ba498b3e52e16916205: Beginning new tablet copy session on tablet 3b3fa060e8d64911bec810ec07a358d2 from peer 272b2179c1b4476090a89880df2a8baa at {username='slave'} at 127.9.176.199:56005: session id = 272b2179c1b4476090a89880df2a8baa-3b3fa060e8d64911bec810ec07a358d2
I20240502 01:54:33.028265 17654 tablet_copy_source_session.cc:205] T 3b3fa060e8d64911bec810ec07a358d2 P 1b50a031f2784ba498b3e52e16916205: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:33.029101 17632 raft_consensus.cc:2932] T 37eda67598fe491bbb29aa62b2c4bec7 P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } } }
I20240502 01:54:33.038707 20699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b3fa060e8d64911bec810ec07a358d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:33.059778 20744 tablet_copy_client.cc:308] T 768e7cbf63a94d3a815ac77a4d0acf41 P 272b2179c1b4476090a89880df2a8baa: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.195:37357
I20240502 01:54:33.067389 20699 tablet_copy_client.cc:783] T 3b3fa060e8d64911bec810ec07a358d2 P 272b2179c1b4476090a89880df2a8baa: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:33.068105 20699 tablet_copy_client.cc:647] T 3b3fa060e8d64911bec810ec07a358d2 P 272b2179c1b4476090a89880df2a8baa: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:33.068830 20708 tablet_copy_client.cc:308] T 91e0e82526c0489a8908e250bf3c851f P 29806d3d37354c169484f6db368f8798: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.194:45617
I20240502 01:54:33.071861 20699 tablet_copy_client.cc:523] T 3b3fa060e8d64911bec810ec07a358d2 P 272b2179c1b4476090a89880df2a8baa: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:33.079000 20699 tablet_bootstrap.cc:492] T 3b3fa060e8d64911bec810ec07a358d2 P 272b2179c1b4476090a89880df2a8baa: Bootstrap starting.
I20240502 01:54:33.105531 17654 tablet_copy_service.cc:140] P 1b50a031f2784ba498b3e52e16916205: Received BeginTabletCopySession request for tablet 91e0e82526c0489a8908e250bf3c851f from peer 29806d3d37354c169484f6db368f8798 ({username='slave'} at 127.9.176.197:33725)
I20240502 01:54:33.105765 17654 tablet_copy_service.cc:161] P 1b50a031f2784ba498b3e52e16916205: Beginning new tablet copy session on tablet 91e0e82526c0489a8908e250bf3c851f from peer 29806d3d37354c169484f6db368f8798 at {username='slave'} at 127.9.176.197:33725: session id = 29806d3d37354c169484f6db368f8798-91e0e82526c0489a8908e250bf3c851f
I20240502 01:54:33.110074 17654 tablet_copy_source_session.cc:205] T 91e0e82526c0489a8908e250bf3c851f P 1b50a031f2784ba498b3e52e16916205: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:33.115051 17766 consensus_queue.cc:237] T 6da59c0626c64a5988e39a831c42c268 P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } }
I20240502 01:54:33.119818 20708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91e0e82526c0489a8908e250bf3c851f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:33.120488 17899 raft_consensus.cc:1261] T 6da59c0626c64a5988e39a831c42c268 P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:33.121460 20699 log.cc:824] T 3b3fa060e8d64911bec810ec07a358d2 P 272b2179c1b4476090a89880df2a8baa: Log is configured to *not* fsync() on all Append() calls
I20240502 01:54:33.122051 20618 consensus_queue.cc:1035] T 6da59c0626c64a5988e39a831c42c268 P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:33.122851 17497 raft_consensus.cc:1261] T 6da59c0626c64a5988e39a831c42c268 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:33.123517 20618 consensus_queue.cc:1035] T 6da59c0626c64a5988e39a831c42c268 P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 01:54:33.125047 17700 consensus_peers.cc:485] T 6da59c0626c64a5988e39a831c42c268 P 84d3a4b174c34e02aedab1e620731359 -> Peer 29806d3d37354c169484f6db368f8798 (127.9.176.197:35011): Couldn't send request to peer 29806d3d37354c169484f6db368f8798. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6da59c0626c64a5988e39a831c42c268. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:33.126153 20618 raft_consensus.cc:2932] T 6da59c0626c64a5988e39a831c42c268 P 84d3a4b174c34e02aedab1e620731359 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } } }
I20240502 01:54:33.128396 17498 raft_consensus.cc:2932] T 6da59c0626c64a5988e39a831c42c268 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } } }
I20240502 01:54:33.129866 17899 raft_consensus.cc:2932] T 6da59c0626c64a5988e39a831c42c268 P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } } }
I20240502 01:54:33.142580 20732 tablet_copy_client.cc:308] T 6da59c0626c64a5988e39a831c42c268 P 29806d3d37354c169484f6db368f8798: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.195:37357
I20240502 01:54:33.145287 17350 catalog_manager.cc:5484] T 6da59c0626c64a5988e39a831c42c268 P da3584f429104d23a1a8899b35cc2138 reported cstate change: config changed from index 4 to 5, NON_VOTER 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) added. New cstate: current_term: 1 leader_uuid: "84d3a4b174c34e02aedab1e620731359" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } } }
I20240502 01:54:33.156140 17789 tablet_copy_service.cc:140] P 84d3a4b174c34e02aedab1e620731359: Received BeginTabletCopySession request for tablet 768e7cbf63a94d3a815ac77a4d0acf41 from peer 272b2179c1b4476090a89880df2a8baa ({username='slave'} at 127.9.176.199:46159)
I20240502 01:54:33.156436 17789 tablet_copy_service.cc:161] P 84d3a4b174c34e02aedab1e620731359: Beginning new tablet copy session on tablet 768e7cbf63a94d3a815ac77a4d0acf41 from peer 272b2179c1b4476090a89880df2a8baa at {username='slave'} at 127.9.176.199:46159: session id = 272b2179c1b4476090a89880df2a8baa-768e7cbf63a94d3a815ac77a4d0acf41
I20240502 01:54:33.161052 17789 tablet_copy_source_session.cc:205] T 768e7cbf63a94d3a815ac77a4d0acf41 P 84d3a4b174c34e02aedab1e620731359: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:33.165360 17789 tablet_copy_service.cc:140] P 84d3a4b174c34e02aedab1e620731359: Received BeginTabletCopySession request for tablet 6da59c0626c64a5988e39a831c42c268 from peer 29806d3d37354c169484f6db368f8798 ({username='slave'} at 127.9.176.197:36787)
I20240502 01:54:33.165594 17789 tablet_copy_service.cc:161] P 84d3a4b174c34e02aedab1e620731359: Beginning new tablet copy session on tablet 6da59c0626c64a5988e39a831c42c268 from peer 29806d3d37354c169484f6db368f8798 at {username='slave'} at 127.9.176.197:36787: session id = 29806d3d37354c169484f6db368f8798-6da59c0626c64a5988e39a831c42c268
I20240502 01:54:33.165637 20744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 768e7cbf63a94d3a815ac77a4d0acf41. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:33.170099 17789 tablet_copy_source_session.cc:205] T 6da59c0626c64a5988e39a831c42c268 P 84d3a4b174c34e02aedab1e620731359: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:33.171695 20732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6da59c0626c64a5988e39a831c42c268. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:33.172689 17498 consensus_queue.cc:237] T 37eda67598fe491bbb29aa62b2c4bec7 P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } }
I20240502 01:54:33.174903 20744 tablet_copy_client.cc:783] T 768e7cbf63a94d3a815ac77a4d0acf41 P 272b2179c1b4476090a89880df2a8baa: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:33.175242 20744 tablet_copy_client.cc:647] T 768e7cbf63a94d3a815ac77a4d0acf41 P 272b2179c1b4476090a89880df2a8baa: tablet copy: Starting download of 1 WAL segments...
W20240502 01:54:33.177253 17432 consensus_peers.cc:485] T 37eda67598fe491bbb29aa62b2c4bec7 P 311f765f59754bdcb4eadc04e14bd4ea -> Peer 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198:36865): Couldn't send request to peer 4b7c382c9edd41babdd1a1afc904a813. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 37eda67598fe491bbb29aa62b2c4bec7. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:33.178282 17766 raft_consensus.cc:1261] T 37eda67598fe491bbb29aa62b2c4bec7 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Refusing update from remote peer 311f765f59754bdcb4eadc04e14bd4ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:33.179239 20465 consensus_queue.cc:1035] T 37eda67598fe491bbb29aa62b2c4bec7 P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:33.183779 20585 raft_consensus.cc:2932] T 37eda67598fe491bbb29aa62b2c4bec7 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } } }
I20240502 01:54:33.186391 20708 tablet_copy_client.cc:783] T 91e0e82526c0489a8908e250bf3c851f P 29806d3d37354c169484f6db368f8798: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:33.186852 20708 tablet_copy_client.cc:647] T 91e0e82526c0489a8908e250bf3c851f P 29806d3d37354c169484f6db368f8798: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:33.188721 17766 raft_consensus.cc:2932] T 37eda67598fe491bbb29aa62b2c4bec7 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } } }
I20240502 01:54:33.191190 20708 tablet_copy_client.cc:523] T 91e0e82526c0489a8908e250bf3c851f P 29806d3d37354c169484f6db368f8798: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:33.207058 17350 catalog_manager.cc:5484] T 37eda67598fe491bbb29aa62b2c4bec7 P 84d3a4b174c34e02aedab1e620731359 reported cstate change: config changed from index 4 to 5, NON_VOTER 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) added. New cstate: current_term: 1 leader_uuid: "311f765f59754bdcb4eadc04e14bd4ea" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } } }
I20240502 01:54:33.213738 20732 tablet_copy_client.cc:783] T 6da59c0626c64a5988e39a831c42c268 P 29806d3d37354c169484f6db368f8798: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:33.215126 20732 tablet_copy_client.cc:647] T 6da59c0626c64a5988e39a831c42c268 P 29806d3d37354c169484f6db368f8798: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:33.219959 20732 tablet_copy_client.cc:523] T 6da59c0626c64a5988e39a831c42c268 P 29806d3d37354c169484f6db368f8798: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:33.224503 17633 raft_consensus.cc:1261] T 37eda67598fe491bbb29aa62b2c4bec7 P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Refusing update from remote peer 311f765f59754bdcb4eadc04e14bd4ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:33.226202 20465 consensus_queue.cc:1035] T 37eda67598fe491bbb29aa62b2c4bec7 P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:33.228127 17633 raft_consensus.cc:2932] T 37eda67598fe491bbb29aa62b2c4bec7 P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } } }
W20240502 01:54:33.229878 17433 consensus_peers.cc:485] T 37eda67598fe491bbb29aa62b2c4bec7 P 311f765f59754bdcb4eadc04e14bd4ea -> Peer 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200:37831): Couldn't send request to peer 94931f4b51c9493cb3d4264ce99f5e76. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 37eda67598fe491bbb29aa62b2c4bec7. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:33.230979 20744 tablet_copy_client.cc:523] T 768e7cbf63a94d3a815ac77a4d0acf41 P 272b2179c1b4476090a89880df2a8baa: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:33.240422 20732 tablet_bootstrap.cc:492] T 6da59c0626c64a5988e39a831c42c268 P 29806d3d37354c169484f6db368f8798: Bootstrap starting.
I20240502 01:54:33.245025 20744 tablet_bootstrap.cc:492] T 768e7cbf63a94d3a815ac77a4d0acf41 P 272b2179c1b4476090a89880df2a8baa: Bootstrap starting.
I20240502 01:54:33.246686 20783 ts_tablet_manager.cc:902] T 3bdd49939e994240a6d15dbd11559ff4 P 94931f4b51c9493cb3d4264ce99f5e76: Initiating tablet copy from peer da3584f429104d23a1a8899b35cc2138 (127.9.176.196:46381)
I20240502 01:54:33.267858 20708 tablet_bootstrap.cc:492] T 91e0e82526c0489a8908e250bf3c851f P 29806d3d37354c169484f6db368f8798: Bootstrap starting.
I20240502 01:54:33.268151 17766 consensus_queue.cc:237] T 768e7cbf63a94d3a815ac77a4d0acf41 P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } }
W20240502 01:54:33.272462 17700 consensus_peers.cc:485] T 768e7cbf63a94d3a815ac77a4d0acf41 P 84d3a4b174c34e02aedab1e620731359 -> Peer 272b2179c1b4476090a89880df2a8baa (127.9.176.199:42269): Couldn't send request to peer 272b2179c1b4476090a89880df2a8baa. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:33.272854 17899 raft_consensus.cc:1261] T 768e7cbf63a94d3a815ac77a4d0acf41 P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:33.273491 20618 consensus_queue.cc:1035] T 768e7cbf63a94d3a815ac77a4d0acf41 P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:33.275765 17633 raft_consensus.cc:1261] T 768e7cbf63a94d3a815ac77a4d0acf41 P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:33.276161 20618 raft_consensus.cc:2932] T 768e7cbf63a94d3a815ac77a4d0acf41 P 84d3a4b174c34e02aedab1e620731359 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } } }
I20240502 01:54:33.277383 20790 ts_tablet_manager.cc:902] T 05b67d9912f047bfb6a2bed90b82ae15 P 4b7c382c9edd41babdd1a1afc904a813: Initiating tablet copy from peer da3584f429104d23a1a8899b35cc2138 (127.9.176.196:46381)
I20240502 01:54:33.279994 20293 consensus_queue.cc:1035] T 768e7cbf63a94d3a815ac77a4d0acf41 P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:33.281996 17633 raft_consensus.cc:2932] T 768e7cbf63a94d3a815ac77a4d0acf41 P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } } }
I20240502 01:54:33.285789 17900 raft_consensus.cc:2932] T 768e7cbf63a94d3a815ac77a4d0acf41 P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } } }
I20240502 01:54:33.292991 17348 catalog_manager.cc:5484] T 768e7cbf63a94d3a815ac77a4d0acf41 P da3584f429104d23a1a8899b35cc2138 reported cstate change: config changed from index 4 to 5, NON_VOTER 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) added. New cstate: current_term: 1 leader_uuid: "84d3a4b174c34e02aedab1e620731359" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } } }
W20240502 01:54:33.304760 17701 consensus_peers.cc:485] T 6da59c0626c64a5988e39a831c42c268 P 84d3a4b174c34e02aedab1e620731359 -> Peer 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200:37831): Couldn't send request to peer 94931f4b51c9493cb3d4264ce99f5e76. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6da59c0626c64a5988e39a831c42c268. This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:33.324911 17700 consensus_peers.cc:485] T 768e7cbf63a94d3a815ac77a4d0acf41 P 84d3a4b174c34e02aedab1e620731359 -> Peer 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198:36865): Couldn't send request to peer 4b7c382c9edd41babdd1a1afc904a813. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 768e7cbf63a94d3a815ac77a4d0acf41. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:33.338796 20808 ts_tablet_manager.cc:902] T c91f74f46b04425aa5393515cb9f781c P 29806d3d37354c169484f6db368f8798: Initiating tablet copy from peer 84d3a4b174c34e02aedab1e620731359 (127.9.176.195:37357)
I20240502 01:54:33.348346 17766 consensus_queue.cc:237] T 2045e08a200148c69326351f38fd484d P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } }
I20240502 01:54:33.357259 20699 tablet_bootstrap.cc:492] T 3b3fa060e8d64911bec810ec07a358d2 P 272b2179c1b4476090a89880df2a8baa: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:33.358078 20699 tablet_bootstrap.cc:492] T 3b3fa060e8d64911bec810ec07a358d2 P 272b2179c1b4476090a89880df2a8baa: Bootstrap complete.
I20240502 01:54:33.358635 20699 ts_tablet_manager.cc:1367] T 3b3fa060e8d64911bec810ec07a358d2 P 272b2179c1b4476090a89880df2a8baa: Time spent bootstrapping tablet: real 0.280s	user 0.082s	sys 0.048s
I20240502 01:54:33.366966 20783 tablet_copy_client.cc:308] T 3bdd49939e994240a6d15dbd11559ff4 P 94931f4b51c9493cb3d4264ce99f5e76: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.196:46381
I20240502 01:54:33.368697 20708 log.cc:824] T 91e0e82526c0489a8908e250bf3c851f P 29806d3d37354c169484f6db368f8798: Log is configured to *not* fsync() on all Append() calls
I20240502 01:54:33.369896 20699 raft_consensus.cc:348] T 3b3fa060e8d64911bec810ec07a358d2 P 272b2179c1b4476090a89880df2a8baa [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } }
I20240502 01:54:33.370424 20699 raft_consensus.cc:729] T 3b3fa060e8d64911bec810ec07a358d2 P 272b2179c1b4476090a89880df2a8baa [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 272b2179c1b4476090a89880df2a8baa, State: Initialized, Role: LEARNER
I20240502 01:54:33.371014 20699 consensus_queue.cc:260] T 3b3fa060e8d64911bec810ec07a358d2 P 272b2179c1b4476090a89880df2a8baa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } }
I20240502 01:54:33.418190 17633 raft_consensus.cc:1261] T 2045e08a200148c69326351f38fd484d P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 01:54:33.418845 17700 consensus_peers.cc:485] T 2045e08a200148c69326351f38fd484d P 84d3a4b174c34e02aedab1e620731359 -> Peer 29806d3d37354c169484f6db368f8798 (127.9.176.197:35011): Couldn't send request to peer 29806d3d37354c169484f6db368f8798. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2045e08a200148c69326351f38fd484d. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:33.419706 20293 consensus_queue.cc:1035] T 2045e08a200148c69326351f38fd484d P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:33.420794 17899 raft_consensus.cc:1261] T 2045e08a200148c69326351f38fd484d P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:33.421698 20618 consensus_queue.cc:1035] T 2045e08a200148c69326351f38fd484d P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:33.428058 20808 tablet_copy_client.cc:308] T c91f74f46b04425aa5393515cb9f781c P 29806d3d37354c169484f6db368f8798: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.195:37357
I20240502 01:54:33.429370 17789 tablet_copy_service.cc:140] P 84d3a4b174c34e02aedab1e620731359: Received BeginTabletCopySession request for tablet c91f74f46b04425aa5393515cb9f781c from peer 29806d3d37354c169484f6db368f8798 ({username='slave'} at 127.9.176.197:36787)
I20240502 01:54:33.429550 17789 tablet_copy_service.cc:161] P 84d3a4b174c34e02aedab1e620731359: Beginning new tablet copy session on tablet c91f74f46b04425aa5393515cb9f781c from peer 29806d3d37354c169484f6db368f8798 at {username='slave'} at 127.9.176.197:36787: session id = 29806d3d37354c169484f6db368f8798-c91f74f46b04425aa5393515cb9f781c
I20240502 01:54:33.433310 17789 tablet_copy_source_session.cc:205] T c91f74f46b04425aa5393515cb9f781c P 84d3a4b174c34e02aedab1e620731359: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:33.434855 20808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c91f74f46b04425aa5393515cb9f781c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:33.437357 17632 consensus_queue.cc:237] T b40f968655774ce887dff8b7f4651501 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } }
I20240502 01:54:33.450313 17922 tablet_copy_service.cc:140] P da3584f429104d23a1a8899b35cc2138: Received BeginTabletCopySession request for tablet 3bdd49939e994240a6d15dbd11559ff4 from peer 94931f4b51c9493cb3d4264ce99f5e76 ({username='slave'} at 127.9.176.200:36935)
I20240502 01:54:33.450619 17922 tablet_copy_service.cc:161] P da3584f429104d23a1a8899b35cc2138: Beginning new tablet copy session on tablet 3bdd49939e994240a6d15dbd11559ff4 from peer 94931f4b51c9493cb3d4264ce99f5e76 at {username='slave'} at 127.9.176.200:36935: session id = 94931f4b51c9493cb3d4264ce99f5e76-3bdd49939e994240a6d15dbd11559ff4
I20240502 01:54:33.455080 17922 tablet_copy_source_session.cc:205] T 3bdd49939e994240a6d15dbd11559ff4 P da3584f429104d23a1a8899b35cc2138: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:33.456179 20808 tablet_copy_client.cc:783] T c91f74f46b04425aa5393515cb9f781c P 29806d3d37354c169484f6db368f8798: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:33.456840 20808 tablet_copy_client.cc:647] T c91f74f46b04425aa5393515cb9f781c P 29806d3d37354c169484f6db368f8798: tablet copy: Starting download of 1 WAL segments...
W20240502 01:54:33.463398 17565 consensus_peers.cc:485] T b40f968655774ce887dff8b7f4651501 P 1b50a031f2784ba498b3e52e16916205 -> Peer 272b2179c1b4476090a89880df2a8baa (127.9.176.199:42269): Couldn't send request to peer 272b2179c1b4476090a89880df2a8baa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b40f968655774ce887dff8b7f4651501. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:33.463680 20808 tablet_copy_client.cc:523] T c91f74f46b04425aa5393515cb9f781c P 29806d3d37354c169484f6db368f8798: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:33.465011 17766 raft_consensus.cc:1261] T b40f968655774ce887dff8b7f4651501 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:33.465574 17900 raft_consensus.cc:1261] T b40f968655774ce887dff8b7f4651501 P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:33.466073 20628 consensus_queue.cc:1035] T b40f968655774ce887dff8b7f4651501 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:33.467044 20628 consensus_queue.cc:1035] T b40f968655774ce887dff8b7f4651501 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:33.474555 20783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bdd49939e994240a6d15dbd11559ff4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:33.482365 20618 raft_consensus.cc:2932] T 2045e08a200148c69326351f38fd484d P 84d3a4b174c34e02aedab1e620731359 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 29806d3d37354c169484f6db368f8798 (127.9.176.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } } }
I20240502 01:54:33.483320 17898 raft_consensus.cc:2932] T 2045e08a200148c69326351f38fd484d P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 29806d3d37354c169484f6db368f8798 (127.9.176.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } } }
I20240502 01:54:33.487978 20808 tablet_bootstrap.cc:492] T c91f74f46b04425aa5393515cb9f781c P 29806d3d37354c169484f6db368f8798: Bootstrap starting.
I20240502 01:54:33.495599 17355 catalog_manager.cc:5484] T 2045e08a200148c69326351f38fd484d P da3584f429104d23a1a8899b35cc2138 reported cstate change: config changed from index -1 to 4, NON_VOTER 29806d3d37354c169484f6db368f8798 (127.9.176.197) added. New cstate: current_term: 1 leader_uuid: "84d3a4b174c34e02aedab1e620731359" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } } }
I20240502 01:54:33.501492 17900 raft_consensus.cc:2932] T b40f968655774ce887dff8b7f4651501 P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 272b2179c1b4476090a89880df2a8baa (127.9.176.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } } }
I20240502 01:54:33.510833 17348 catalog_manager.cc:5484] T b40f968655774ce887dff8b7f4651501 P da3584f429104d23a1a8899b35cc2138 reported cstate change: config changed from index -1 to 4, NON_VOTER 272b2179c1b4476090a89880df2a8baa (127.9.176.199) added. New cstate: current_term: 1 leader_uuid: "1b50a031f2784ba498b3e52e16916205" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } } }
I20240502 01:54:33.511857 20635 raft_consensus.cc:2932] T b40f968655774ce887dff8b7f4651501 P 1b50a031f2784ba498b3e52e16916205 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 272b2179c1b4476090a89880df2a8baa (127.9.176.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } } }
I20240502 01:54:33.513895 17632 raft_consensus.cc:2932] T 2045e08a200148c69326351f38fd484d P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 29806d3d37354c169484f6db368f8798 (127.9.176.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } } }
I20240502 01:54:33.540431 20790 tablet_copy_client.cc:308] T 05b67d9912f047bfb6a2bed90b82ae15 P 4b7c382c9edd41babdd1a1afc904a813: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.196:46381
I20240502 01:54:33.564306 17766 raft_consensus.cc:2932] T b40f968655774ce887dff8b7f4651501 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 272b2179c1b4476090a89880df2a8baa (127.9.176.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } } }
I20240502 01:54:33.567396 20699 ts_tablet_manager.cc:1398] T 3b3fa060e8d64911bec810ec07a358d2 P 272b2179c1b4476090a89880df2a8baa: Time spent starting tablet: real 0.209s	user 0.024s	sys 0.054s
I20240502 01:54:33.576985 17633 consensus_queue.cc:237] T df16e178e04040699201f325ce9e2979 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } }
I20240502 01:54:33.583827 17654 tablet_copy_service.cc:342] P 1b50a031f2784ba498b3e52e16916205: Request end of tablet copy session 272b2179c1b4476090a89880df2a8baa-3b3fa060e8d64911bec810ec07a358d2 received from {username='slave'} at 127.9.176.199:56005
I20240502 01:54:33.584038 17654 tablet_copy_service.cc:433] P 1b50a031f2784ba498b3e52e16916205: ending tablet copy session 272b2179c1b4476090a89880df2a8baa-3b3fa060e8d64911bec810ec07a358d2 on tablet 3b3fa060e8d64911bec810ec07a358d2 with peer 272b2179c1b4476090a89880df2a8baa
I20240502 01:54:33.605752 17497 raft_consensus.cc:1261] T df16e178e04040699201f325ce9e2979 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 01:54:33.607100 17565 consensus_peers.cc:485] T df16e178e04040699201f325ce9e2979 P 1b50a031f2784ba498b3e52e16916205 -> Peer 272b2179c1b4476090a89880df2a8baa (127.9.176.199:42269): Couldn't send request to peer 272b2179c1b4476090a89880df2a8baa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: df16e178e04040699201f325ce9e2979. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:33.607723 20628 consensus_queue.cc:1035] T df16e178e04040699201f325ce9e2979 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:33.615654 17922 tablet_copy_service.cc:140] P da3584f429104d23a1a8899b35cc2138: Received BeginTabletCopySession request for tablet 05b67d9912f047bfb6a2bed90b82ae15 from peer 4b7c382c9edd41babdd1a1afc904a813 ({username='slave'} at 127.9.176.198:45579)
I20240502 01:54:33.615903 17922 tablet_copy_service.cc:161] P da3584f429104d23a1a8899b35cc2138: Beginning new tablet copy session on tablet 05b67d9912f047bfb6a2bed90b82ae15 from peer 4b7c382c9edd41babdd1a1afc904a813 at {username='slave'} at 127.9.176.198:45579: session id = 4b7c382c9edd41babdd1a1afc904a813-05b67d9912f047bfb6a2bed90b82ae15
I20240502 01:54:33.619021 17922 tablet_copy_source_session.cc:205] T 05b67d9912f047bfb6a2bed90b82ae15 P da3584f429104d23a1a8899b35cc2138: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:33.631045 17746 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:54:33.640398 19585 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:33.653064 19718 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:33.654403 20790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05b67d9912f047bfb6a2bed90b82ae15. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:33.661007 19446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:33.661386 17766 raft_consensus.cc:1261] T df16e178e04040699201f325ce9e2979 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:33.663781 20628 consensus_queue.cc:1035] T df16e178e04040699201f325ce9e2979 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:33.666744 19718 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:33.672426 19852 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:33.684674 17613 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:54:33.688448 17613 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:54:33.689549 20635 raft_consensus.cc:2932] T df16e178e04040699201f325ce9e2979 P 1b50a031f2784ba498b3e52e16916205 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 272b2179c1b4476090a89880df2a8baa (127.9.176.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } } }
I20240502 01:54:33.712180 20873 ts_tablet_manager.cc:902] T 37eda67598fe491bbb29aa62b2c4bec7 P 94931f4b51c9493cb3d4264ce99f5e76: Initiating tablet copy from peer 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193:35397)
I20240502 01:54:33.714964 17497 raft_consensus.cc:2932] T df16e178e04040699201f325ce9e2979 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 272b2179c1b4476090a89880df2a8baa (127.9.176.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } } }
I20240502 01:54:33.715718 17350 catalog_manager.cc:5484] T df16e178e04040699201f325ce9e2979 P 1b50a031f2784ba498b3e52e16916205 reported cstate change: config changed from index -1 to 4, NON_VOTER 272b2179c1b4476090a89880df2a8baa (127.9.176.199) added. New cstate: current_term: 1 leader_uuid: "1b50a031f2784ba498b3e52e16916205" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:33.717288 17880 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:54:33.743580 17766 raft_consensus.cc:2932] T df16e178e04040699201f325ce9e2979 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 272b2179c1b4476090a89880df2a8baa (127.9.176.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } } }
I20240502 01:54:33.762290 17477 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:54:33.773782 19446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:33.777869 19585 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:33.801064 17633 consensus_queue.cc:237] T 2c78951695284b1aaab411c0a041c28a P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } }
I20240502 01:54:33.824486 20783 tablet_copy_client.cc:783] T 3bdd49939e994240a6d15dbd11559ff4 P 94931f4b51c9493cb3d4264ce99f5e76: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:33.825620 20783 tablet_copy_client.cc:647] T 3bdd49939e994240a6d15dbd11559ff4 P 94931f4b51c9493cb3d4264ce99f5e76: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:33.825716 17746 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:54:33.832062 20878 ts_tablet_manager.cc:902] T 768e7cbf63a94d3a815ac77a4d0acf41 P 4b7c382c9edd41babdd1a1afc904a813: Initiating tablet copy from peer 84d3a4b174c34e02aedab1e620731359 (127.9.176.195:37357)
I20240502 01:54:33.839556 20881 ts_tablet_manager.cc:902] T 37eda67598fe491bbb29aa62b2c4bec7 P 4b7c382c9edd41babdd1a1afc904a813: Initiating tablet copy from peer 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193:35397)
I20240502 01:54:33.847044 20699 ts_tablet_manager.cc:902] T b40f968655774ce887dff8b7f4651501 P 272b2179c1b4476090a89880df2a8baa: Initiating tablet copy from peer 1b50a031f2784ba498b3e52e16916205 (127.9.176.194:45617)
I20240502 01:54:33.854557 17497 raft_consensus.cc:1261] T 2c78951695284b1aaab411c0a041c28a P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:33.854655 17767 raft_consensus.cc:1261] T 2c78951695284b1aaab411c0a041c28a P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:33.864986 17477 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:54:33.867771 20699 tablet_copy_client.cc:308] T b40f968655774ce887dff8b7f4651501 P 272b2179c1b4476090a89880df2a8baa: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.194:45617
I20240502 01:54:33.835489 20879 ts_tablet_manager.cc:902] T 6da59c0626c64a5988e39a831c42c268 P 94931f4b51c9493cb3d4264ce99f5e76: Initiating tablet copy from peer 84d3a4b174c34e02aedab1e620731359 (127.9.176.195:37357)
I20240502 01:54:33.873103 20783 tablet_copy_client.cc:523] T 3bdd49939e994240a6d15dbd11559ff4 P 94931f4b51c9493cb3d4264ce99f5e76: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:33.866480 20873 tablet_copy_client.cc:308] T 37eda67598fe491bbb29aa62b2c4bec7 P 94931f4b51c9493cb3d4264ce99f5e76: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.193:35397
I20240502 01:54:33.893275 20744 tablet_bootstrap.cc:492] T 768e7cbf63a94d3a815ac77a4d0acf41 P 272b2179c1b4476090a89880df2a8baa: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:33.903242 19852 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:33.908051 20744 tablet_bootstrap.cc:492] T 768e7cbf63a94d3a815ac77a4d0acf41 P 272b2179c1b4476090a89880df2a8baa: Bootstrap complete.
I20240502 01:54:33.908419 20744 ts_tablet_manager.cc:1367] T 768e7cbf63a94d3a815ac77a4d0acf41 P 272b2179c1b4476090a89880df2a8baa: Time spent bootstrapping tablet: real 0.664s	user 0.063s	sys 0.004s
I20240502 01:54:33.909557 20744 raft_consensus.cc:348] T 768e7cbf63a94d3a815ac77a4d0acf41 P 272b2179c1b4476090a89880df2a8baa [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } }
I20240502 01:54:33.909916 20744 raft_consensus.cc:729] T 768e7cbf63a94d3a815ac77a4d0acf41 P 272b2179c1b4476090a89880df2a8baa [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 272b2179c1b4476090a89880df2a8baa, State: Initialized, Role: LEARNER
I20240502 01:54:33.910156 20744 consensus_queue.cc:260] T 768e7cbf63a94d3a815ac77a4d0acf41 P 272b2179c1b4476090a89880df2a8baa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } }
I20240502 01:54:33.910660 17880 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:54:33.916059 17520 tablet_copy_service.cc:140] P 311f765f59754bdcb4eadc04e14bd4ea: Received BeginTabletCopySession request for tablet 37eda67598fe491bbb29aa62b2c4bec7 from peer 94931f4b51c9493cb3d4264ce99f5e76 ({username='slave'} at 127.9.176.200:46111)
I20240502 01:54:33.916368 17520 tablet_copy_service.cc:161] P 311f765f59754bdcb4eadc04e14bd4ea: Beginning new tablet copy session on tablet 37eda67598fe491bbb29aa62b2c4bec7 from peer 94931f4b51c9493cb3d4264ce99f5e76 at {username='slave'} at 127.9.176.200:46111: session id = 94931f4b51c9493cb3d4264ce99f5e76-37eda67598fe491bbb29aa62b2c4bec7
I20240502 01:54:33.923219 17520 tablet_copy_source_session.cc:205] T 37eda67598fe491bbb29aa62b2c4bec7 P 311f765f59754bdcb4eadc04e14bd4ea: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:33.930408 17654 tablet_copy_service.cc:140] P 1b50a031f2784ba498b3e52e16916205: Received BeginTabletCopySession request for tablet b40f968655774ce887dff8b7f4651501 from peer 272b2179c1b4476090a89880df2a8baa ({username='slave'} at 127.9.176.199:56005)
I20240502 01:54:33.930681 17654 tablet_copy_service.cc:161] P 1b50a031f2784ba498b3e52e16916205: Beginning new tablet copy session on tablet b40f968655774ce887dff8b7f4651501 from peer 272b2179c1b4476090a89880df2a8baa at {username='slave'} at 127.9.176.199:56005: session id = 272b2179c1b4476090a89880df2a8baa-b40f968655774ce887dff8b7f4651501
I20240502 01:54:33.934094 17654 tablet_copy_source_session.cc:205] T b40f968655774ce887dff8b7f4651501 P 1b50a031f2784ba498b3e52e16916205: Tablet Copy: opened 0 blocks and 1 log segments
W20240502 01:54:33.935592 17565 consensus_peers.cc:485] T 2c78951695284b1aaab411c0a041c28a P 1b50a031f2784ba498b3e52e16916205 -> Peer 272b2179c1b4476090a89880df2a8baa (127.9.176.199:42269): Couldn't send request to peer 272b2179c1b4476090a89880df2a8baa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2c78951695284b1aaab411c0a041c28a. This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:33.935735 17565 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.176.199:42269, user_credentials={real_user=slave}} blocked reactor thread for 82197.8us
I20240502 01:54:33.963181 20877 ts_tablet_manager.cc:902] T 2045e08a200148c69326351f38fd484d P 29806d3d37354c169484f6db368f8798: Initiating tablet copy from peer 84d3a4b174c34e02aedab1e620731359 (127.9.176.195:37357)
I20240502 01:54:33.973754 20879 tablet_copy_client.cc:308] T 6da59c0626c64a5988e39a831c42c268 P 94931f4b51c9493cb3d4264ce99f5e76: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.195:37357
I20240502 01:54:33.976799 20635 consensus_queue.cc:1035] T 2c78951695284b1aaab411c0a041c28a P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:33.977535 20635 consensus_queue.cc:1035] T 2c78951695284b1aaab411c0a041c28a P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:33.988703 20881 tablet_copy_client.cc:308] T 37eda67598fe491bbb29aa62b2c4bec7 P 4b7c382c9edd41babdd1a1afc904a813: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.193:35397
I20240502 01:54:33.990437 20732 tablet_bootstrap.cc:492] T 6da59c0626c64a5988e39a831c42c268 P 29806d3d37354c169484f6db368f8798: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:33.990478 17789 tablet_copy_service.cc:140] P 84d3a4b174c34e02aedab1e620731359: Received BeginTabletCopySession request for tablet 6da59c0626c64a5988e39a831c42c268 from peer 94931f4b51c9493cb3d4264ce99f5e76 ({username='slave'} at 127.9.176.200:34957)
I20240502 01:54:33.990660 17789 tablet_copy_service.cc:161] P 84d3a4b174c34e02aedab1e620731359: Beginning new tablet copy session on tablet 6da59c0626c64a5988e39a831c42c268 from peer 94931f4b51c9493cb3d4264ce99f5e76 at {username='slave'} at 127.9.176.200:34957: session id = 94931f4b51c9493cb3d4264ce99f5e76-6da59c0626c64a5988e39a831c42c268
I20240502 01:54:33.991155 20732 tablet_bootstrap.cc:492] T 6da59c0626c64a5988e39a831c42c268 P 29806d3d37354c169484f6db368f8798: Bootstrap complete.
I20240502 01:54:33.991777 20732 ts_tablet_manager.cc:1367] T 6da59c0626c64a5988e39a831c42c268 P 29806d3d37354c169484f6db368f8798: Time spent bootstrapping tablet: real 0.753s	user 0.066s	sys 0.004s
I20240502 01:54:33.993880 17789 tablet_copy_source_session.cc:205] T 6da59c0626c64a5988e39a831c42c268 P 84d3a4b174c34e02aedab1e620731359: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:33.995968 20879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6da59c0626c64a5988e39a831c42c268. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:33.996112 20744 ts_tablet_manager.cc:1398] T 768e7cbf63a94d3a815ac77a4d0acf41 P 272b2179c1b4476090a89880df2a8baa: Time spent starting tablet: real 0.088s	user 0.003s	sys 0.000s
I20240502 01:54:34.003213 20708 tablet_bootstrap.cc:492] T 91e0e82526c0489a8908e250bf3c851f P 29806d3d37354c169484f6db368f8798: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:34.004128 20873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37eda67598fe491bbb29aa62b2c4bec7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:33.960847 20783 tablet_bootstrap.cc:492] T 3bdd49939e994240a6d15dbd11559ff4 P 94931f4b51c9493cb3d4264ce99f5e76: Bootstrap starting.
I20240502 01:54:34.014485 20708 tablet_bootstrap.cc:492] T 91e0e82526c0489a8908e250bf3c851f P 29806d3d37354c169484f6db368f8798: Bootstrap complete.
I20240502 01:54:34.014925 20708 ts_tablet_manager.cc:1367] T 91e0e82526c0489a8908e250bf3c851f P 29806d3d37354c169484f6db368f8798: Time spent bootstrapping tablet: real 0.810s	user 0.057s	sys 0.013s
I20240502 01:54:34.016101 17520 tablet_copy_service.cc:140] P 311f765f59754bdcb4eadc04e14bd4ea: Received BeginTabletCopySession request for tablet 37eda67598fe491bbb29aa62b2c4bec7 from peer 4b7c382c9edd41babdd1a1afc904a813 ({username='slave'} at 127.9.176.198:53399)
I20240502 01:54:34.016300 17520 tablet_copy_service.cc:161] P 311f765f59754bdcb4eadc04e14bd4ea: Beginning new tablet copy session on tablet 37eda67598fe491bbb29aa62b2c4bec7 from peer 4b7c382c9edd41babdd1a1afc904a813 at {username='slave'} at 127.9.176.198:53399: session id = 4b7c382c9edd41babdd1a1afc904a813-37eda67598fe491bbb29aa62b2c4bec7
I20240502 01:54:34.019484 17520 tablet_copy_source_session.cc:205] T 37eda67598fe491bbb29aa62b2c4bec7 P 311f765f59754bdcb4eadc04e14bd4ea: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:34.021739 20881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37eda67598fe491bbb29aa62b2c4bec7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:34.024408 20873 tablet_copy_client.cc:783] T 37eda67598fe491bbb29aa62b2c4bec7 P 94931f4b51c9493cb3d4264ce99f5e76: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:34.024717 20873 tablet_copy_client.cc:647] T 37eda67598fe491bbb29aa62b2c4bec7 P 94931f4b51c9493cb3d4264ce99f5e76: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:34.029603 17789 tablet_copy_service.cc:342] P 84d3a4b174c34e02aedab1e620731359: Request end of tablet copy session 272b2179c1b4476090a89880df2a8baa-768e7cbf63a94d3a815ac77a4d0acf41 received from {username='slave'} at 127.9.176.199:46159
I20240502 01:54:34.029783 17789 tablet_copy_service.cc:433] P 84d3a4b174c34e02aedab1e620731359: ending tablet copy session 272b2179c1b4476090a89880df2a8baa-768e7cbf63a94d3a815ac77a4d0acf41 on tablet 768e7cbf63a94d3a815ac77a4d0acf41 with peer 272b2179c1b4476090a89880df2a8baa
I20240502 01:54:34.036147 20881 tablet_copy_client.cc:783] T 37eda67598fe491bbb29aa62b2c4bec7 P 4b7c382c9edd41babdd1a1afc904a813: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:34.036578 20881 tablet_copy_client.cc:647] T 37eda67598fe491bbb29aa62b2c4bec7 P 4b7c382c9edd41babdd1a1afc904a813: tablet copy: Starting download of 1 WAL segments...
W20240502 01:54:34.037441 17568 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.176.195:37357, user_credentials={real_user=slave}} blocked reactor thread for 116920us
I20240502 01:54:34.039130 20881 tablet_copy_client.cc:523] T 37eda67598fe491bbb29aa62b2c4bec7 P 4b7c382c9edd41babdd1a1afc904a813: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240502 01:54:34.044780 17835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.176.194:45617, user_credentials={real_user=slave}} blocked reactor thread for 107824us
I20240502 01:54:34.049283 20699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b40f968655774ce887dff8b7f4651501. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:34.057073 20699 tablet_copy_client.cc:783] T b40f968655774ce887dff8b7f4651501 P 272b2179c1b4476090a89880df2a8baa: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:34.057443 20699 tablet_copy_client.cc:647] T b40f968655774ce887dff8b7f4651501 P 272b2179c1b4476090a89880df2a8baa: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:34.062264 20873 tablet_copy_client.cc:523] T 37eda67598fe491bbb29aa62b2c4bec7 P 94931f4b51c9493cb3d4264ce99f5e76: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:34.062950 20881 tablet_bootstrap.cc:492] T 37eda67598fe491bbb29aa62b2c4bec7 P 4b7c382c9edd41babdd1a1afc904a813: Bootstrap starting.
I20240502 01:54:34.070106 20873 tablet_bootstrap.cc:492] T 37eda67598fe491bbb29aa62b2c4bec7 P 94931f4b51c9493cb3d4264ce99f5e76: Bootstrap starting.
I20240502 01:54:34.087416 19738 raft_consensus.cc:1203] T 3b3fa060e8d64911bec810ec07a358d2 P 272b2179c1b4476090a89880df2a8baa [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 01:54:34.096105 20708 raft_consensus.cc:348] T 91e0e82526c0489a8908e250bf3c851f P 29806d3d37354c169484f6db368f8798 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } }
I20240502 01:54:34.097739 20708 raft_consensus.cc:729] T 91e0e82526c0489a8908e250bf3c851f P 29806d3d37354c169484f6db368f8798 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 29806d3d37354c169484f6db368f8798, State: Initialized, Role: LEARNER
I20240502 01:54:34.098502 20708 consensus_queue.cc:260] T 91e0e82526c0489a8908e250bf3c851f P 29806d3d37354c169484f6db368f8798 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } }
I20240502 01:54:34.100684 20708 ts_tablet_manager.cc:1398] T 91e0e82526c0489a8908e250bf3c851f P 29806d3d37354c169484f6db368f8798: Time spent starting tablet: real 0.086s	user 0.006s	sys 0.000s
I20240502 01:54:34.101550 20732 raft_consensus.cc:348] T 6da59c0626c64a5988e39a831c42c268 P 29806d3d37354c169484f6db368f8798 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } }
I20240502 01:54:34.101902 20732 raft_consensus.cc:729] T 6da59c0626c64a5988e39a831c42c268 P 29806d3d37354c169484f6db368f8798 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 29806d3d37354c169484f6db368f8798, State: Initialized, Role: LEARNER
I20240502 01:54:34.102147 20732 consensus_queue.cc:260] T 6da59c0626c64a5988e39a831c42c268 P 29806d3d37354c169484f6db368f8798 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } }
I20240502 01:54:34.105422 20732 ts_tablet_manager.cc:1398] T 6da59c0626c64a5988e39a831c42c268 P 29806d3d37354c169484f6db368f8798: Time spent starting tablet: real 0.086s	user 0.008s	sys 0.000s
I20240502 01:54:34.106565 17789 tablet_copy_service.cc:342] P 84d3a4b174c34e02aedab1e620731359: Request end of tablet copy session 29806d3d37354c169484f6db368f8798-6da59c0626c64a5988e39a831c42c268 received from {username='slave'} at 127.9.176.197:36787
I20240502 01:54:34.106812 17789 tablet_copy_service.cc:433] P 84d3a4b174c34e02aedab1e620731359: ending tablet copy session 29806d3d37354c169484f6db368f8798-6da59c0626c64a5988e39a831c42c268 on tablet 6da59c0626c64a5988e39a831c42c268 with peer 29806d3d37354c169484f6db368f8798
I20240502 01:54:34.104177 20879 tablet_copy_client.cc:783] T 6da59c0626c64a5988e39a831c42c268 P 94931f4b51c9493cb3d4264ce99f5e76: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:34.129559 20879 tablet_copy_client.cc:647] T 6da59c0626c64a5988e39a831c42c268 P 94931f4b51c9493cb3d4264ce99f5e76: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:34.133683 20699 tablet_copy_client.cc:523] T b40f968655774ce887dff8b7f4651501 P 272b2179c1b4476090a89880df2a8baa: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:34.133729 20879 tablet_copy_client.cc:523] T 6da59c0626c64a5988e39a831c42c268 P 94931f4b51c9493cb3d4264ce99f5e76: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:34.138031 20699 tablet_bootstrap.cc:492] T b40f968655774ce887dff8b7f4651501 P 272b2179c1b4476090a89880df2a8baa: Bootstrap starting.
I20240502 01:54:34.143996 20783 log.cc:824] T 3bdd49939e994240a6d15dbd11559ff4 P 94931f4b51c9493cb3d4264ce99f5e76: Log is configured to *not* fsync() on all Append() calls
I20240502 01:54:34.146442 20878 tablet_copy_client.cc:308] T 768e7cbf63a94d3a815ac77a4d0acf41 P 4b7c382c9edd41babdd1a1afc904a813: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.195:37357
I20240502 01:54:34.147199 20744 ts_tablet_manager.cc:902] T df16e178e04040699201f325ce9e2979 P 272b2179c1b4476090a89880df2a8baa: Initiating tablet copy from peer 1b50a031f2784ba498b3e52e16916205 (127.9.176.194:45617)
I20240502 01:54:34.158918 17789 tablet_copy_service.cc:140] P 84d3a4b174c34e02aedab1e620731359: Received BeginTabletCopySession request for tablet 768e7cbf63a94d3a815ac77a4d0acf41 from peer 4b7c382c9edd41babdd1a1afc904a813 ({username='slave'} at 127.9.176.198:55177)
I20240502 01:54:34.159147 17789 tablet_copy_service.cc:161] P 84d3a4b174c34e02aedab1e620731359: Beginning new tablet copy session on tablet 768e7cbf63a94d3a815ac77a4d0acf41 from peer 4b7c382c9edd41babdd1a1afc904a813 at {username='slave'} at 127.9.176.198:55177: session id = 4b7c382c9edd41babdd1a1afc904a813-768e7cbf63a94d3a815ac77a4d0acf41
I20240502 01:54:34.163426 20808 tablet_bootstrap.cc:492] T c91f74f46b04425aa5393515cb9f781c P 29806d3d37354c169484f6db368f8798: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:34.164652 17789 tablet_copy_source_session.cc:205] T 768e7cbf63a94d3a815ac77a4d0acf41 P 84d3a4b174c34e02aedab1e620731359: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:34.165603 20808 tablet_bootstrap.cc:492] T c91f74f46b04425aa5393515cb9f781c P 29806d3d37354c169484f6db368f8798: Bootstrap complete.
I20240502 01:54:34.165984 20808 ts_tablet_manager.cc:1367] T c91f74f46b04425aa5393515cb9f781c P 29806d3d37354c169484f6db368f8798: Time spent bootstrapping tablet: real 0.678s	user 0.044s	sys 0.014s
I20240502 01:54:34.166728 20878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 768e7cbf63a94d3a815ac77a4d0acf41. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:34.164595 20881 log.cc:824] T 37eda67598fe491bbb29aa62b2c4bec7 P 4b7c382c9edd41babdd1a1afc904a813: Log is configured to *not* fsync() on all Append() calls
I20240502 01:54:34.171746 20879 tablet_bootstrap.cc:492] T 6da59c0626c64a5988e39a831c42c268 P 94931f4b51c9493cb3d4264ce99f5e76: Bootstrap starting.
I20240502 01:54:34.171813 20877 tablet_copy_client.cc:308] T 2045e08a200148c69326351f38fd484d P 29806d3d37354c169484f6db368f8798: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.195:37357
I20240502 01:54:34.172955 17789 tablet_copy_service.cc:140] P 84d3a4b174c34e02aedab1e620731359: Received BeginTabletCopySession request for tablet 2045e08a200148c69326351f38fd484d from peer 29806d3d37354c169484f6db368f8798 ({username='slave'} at 127.9.176.197:36787)
I20240502 01:54:34.172897 20808 raft_consensus.cc:348] T c91f74f46b04425aa5393515cb9f781c P 29806d3d37354c169484f6db368f8798 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } }
I20240502 01:54:34.173161 17789 tablet_copy_service.cc:161] P 84d3a4b174c34e02aedab1e620731359: Beginning new tablet copy session on tablet 2045e08a200148c69326351f38fd484d from peer 29806d3d37354c169484f6db368f8798 at {username='slave'} at 127.9.176.197:36787: session id = 29806d3d37354c169484f6db368f8798-2045e08a200148c69326351f38fd484d
I20240502 01:54:34.173238 20808 raft_consensus.cc:729] T c91f74f46b04425aa5393515cb9f781c P 29806d3d37354c169484f6db368f8798 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 29806d3d37354c169484f6db368f8798, State: Initialized, Role: LEARNER
I20240502 01:54:34.173509 20808 consensus_queue.cc:260] T c91f74f46b04425aa5393515cb9f781c P 29806d3d37354c169484f6db368f8798 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } }
I20240502 01:54:34.174304 20808 ts_tablet_manager.cc:1398] T c91f74f46b04425aa5393515cb9f781c P 29806d3d37354c169484f6db368f8798: Time spent starting tablet: real 0.008s	user 0.003s	sys 0.000s
I20240502 01:54:34.175760 20878 tablet_copy_client.cc:783] T 768e7cbf63a94d3a815ac77a4d0acf41 P 4b7c382c9edd41babdd1a1afc904a813: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:34.176122 20878 tablet_copy_client.cc:647] T 768e7cbf63a94d3a815ac77a4d0acf41 P 4b7c382c9edd41babdd1a1afc904a813: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:34.176577 17789 tablet_copy_source_session.cc:205] T 2045e08a200148c69326351f38fd484d P 84d3a4b174c34e02aedab1e620731359: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:34.180620 17788 tablet_copy_service.cc:342] P 84d3a4b174c34e02aedab1e620731359: Request end of tablet copy session 29806d3d37354c169484f6db368f8798-c91f74f46b04425aa5393515cb9f781c received from {username='slave'} at 127.9.176.197:36787
I20240502 01:54:34.180795 17788 tablet_copy_service.cc:433] P 84d3a4b174c34e02aedab1e620731359: ending tablet copy session 29806d3d37354c169484f6db368f8798-c91f74f46b04425aa5393515cb9f781c on tablet c91f74f46b04425aa5393515cb9f781c with peer 29806d3d37354c169484f6db368f8798
I20240502 01:54:34.182262 20877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2045e08a200148c69326351f38fd484d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:34.186693 17654 tablet_copy_service.cc:342] P 1b50a031f2784ba498b3e52e16916205: Request end of tablet copy session 29806d3d37354c169484f6db368f8798-91e0e82526c0489a8908e250bf3c851f received from {username='slave'} at 127.9.176.197:33725
I20240502 01:54:34.186905 17654 tablet_copy_service.cc:433] P 1b50a031f2784ba498b3e52e16916205: ending tablet copy session 29806d3d37354c169484f6db368f8798-91e0e82526c0489a8908e250bf3c851f on tablet 91e0e82526c0489a8908e250bf3c851f with peer 29806d3d37354c169484f6db368f8798
I20240502 01:54:34.195019 20877 tablet_copy_client.cc:783] T 2045e08a200148c69326351f38fd484d P 29806d3d37354c169484f6db368f8798: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:34.195384 20877 tablet_copy_client.cc:647] T 2045e08a200148c69326351f38fd484d P 29806d3d37354c169484f6db368f8798: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:34.214040 20878 tablet_copy_client.cc:523] T 768e7cbf63a94d3a815ac77a4d0acf41 P 4b7c382c9edd41babdd1a1afc904a813: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:34.215461 19467 raft_consensus.cc:1203] T 91e0e82526c0489a8908e250bf3c851f P 29806d3d37354c169484f6db368f8798 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 01:54:34.216845 20877 tablet_copy_client.cc:523] T 2045e08a200148c69326351f38fd484d P 29806d3d37354c169484f6db368f8798: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:34.218158 20878 tablet_bootstrap.cc:492] T 768e7cbf63a94d3a815ac77a4d0acf41 P 4b7c382c9edd41babdd1a1afc904a813: Bootstrap starting.
I20240502 01:54:34.222177 20877 tablet_bootstrap.cc:492] T 2045e08a200148c69326351f38fd484d P 29806d3d37354c169484f6db368f8798: Bootstrap starting.
I20240502 01:54:34.225751 19467 raft_consensus.cc:1203] T c91f74f46b04425aa5393515cb9f781c P 29806d3d37354c169484f6db368f8798 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 01:54:34.232609 17632 consensus_queue.cc:237] T 38a60d7db47149ff90444e5276fc8935 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } }
I20240502 01:54:34.242429 20920 raft_consensus.cc:2932] T 2c78951695284b1aaab411c0a041c28a P 1b50a031f2784ba498b3e52e16916205 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 272b2179c1b4476090a89880df2a8baa (127.9.176.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } } }
I20240502 01:54:34.255251 17766 raft_consensus.cc:2932] T 2c78951695284b1aaab411c0a041c28a P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 272b2179c1b4476090a89880df2a8baa (127.9.176.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } } }
I20240502 01:54:34.260021 20790 tablet_copy_client.cc:783] T 05b67d9912f047bfb6a2bed90b82ae15 P 4b7c382c9edd41babdd1a1afc904a813: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:34.263262 20790 tablet_copy_client.cc:647] T 05b67d9912f047bfb6a2bed90b82ae15 P 4b7c382c9edd41babdd1a1afc904a813: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:34.270188 17355 catalog_manager.cc:5484] T 2c78951695284b1aaab411c0a041c28a P 84d3a4b174c34e02aedab1e620731359 reported cstate change: config changed from index -1 to 4, NON_VOTER 272b2179c1b4476090a89880df2a8baa (127.9.176.199) added. New cstate: current_term: 1 leader_uuid: "1b50a031f2784ba498b3e52e16916205" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } } }
I20240502 01:54:34.278023 17497 raft_consensus.cc:2932] T 2c78951695284b1aaab411c0a041c28a P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 272b2179c1b4476090a89880df2a8baa (127.9.176.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } } }
I20240502 01:54:34.284955 20744 tablet_copy_client.cc:308] T df16e178e04040699201f325ce9e2979 P 272b2179c1b4476090a89880df2a8baa: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.194:45617
I20240502 01:54:34.285763 17654 tablet_copy_service.cc:140] P 1b50a031f2784ba498b3e52e16916205: Received BeginTabletCopySession request for tablet df16e178e04040699201f325ce9e2979 from peer 272b2179c1b4476090a89880df2a8baa ({username='slave'} at 127.9.176.199:56005)
I20240502 01:54:34.289913 20790 tablet_copy_client.cc:523] T 05b67d9912f047bfb6a2bed90b82ae15 P 4b7c382c9edd41babdd1a1afc904a813: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:34.295312 20699 tablet_bootstrap.cc:700] T b40f968655774ce887dff8b7f4651501 P 272b2179c1b4476090a89880df2a8baa: Time spent opening tablet: real 0.156s	user 0.004s	sys 0.000s
I20240502 01:54:34.302999 17654 tablet_copy_service.cc:161] P 1b50a031f2784ba498b3e52e16916205: Beginning new tablet copy session on tablet df16e178e04040699201f325ce9e2979 from peer 272b2179c1b4476090a89880df2a8baa at {username='slave'} at 127.9.176.199:56005: session id = 272b2179c1b4476090a89880df2a8baa-df16e178e04040699201f325ce9e2979
I20240502 01:54:34.307511 17654 tablet_copy_source_session.cc:205] T df16e178e04040699201f325ce9e2979 P 1b50a031f2784ba498b3e52e16916205: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:34.310122 20744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df16e178e04040699201f325ce9e2979. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:34.327368 20744 tablet_copy_client.cc:783] T df16e178e04040699201f325ce9e2979 P 272b2179c1b4476090a89880df2a8baa: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:34.327443 17498 raft_consensus.cc:1261] T 38a60d7db47149ff90444e5276fc8935 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:34.327747 20744 tablet_copy_client.cc:647] T df16e178e04040699201f325ce9e2979 P 272b2179c1b4476090a89880df2a8baa: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:34.328361 20628 consensus_queue.cc:1035] T 38a60d7db47149ff90444e5276fc8935 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:34.331390 17766 raft_consensus.cc:1261] T 38a60d7db47149ff90444e5276fc8935 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:34.333909 20628 consensus_queue.cc:1035] T 38a60d7db47149ff90444e5276fc8935 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:34.345065 20947 ts_tablet_manager.cc:902] T 2c78951695284b1aaab411c0a041c28a P 272b2179c1b4476090a89880df2a8baa: Initiating tablet copy from peer 1b50a031f2784ba498b3e52e16916205 (127.9.176.194:45617)
I20240502 01:54:34.348871 20744 tablet_copy_client.cc:523] T df16e178e04040699201f325ce9e2979 P 272b2179c1b4476090a89880df2a8baa: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:34.353175 20628 raft_consensus.cc:2932] T 38a60d7db47149ff90444e5276fc8935 P 1b50a031f2784ba498b3e52e16916205 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } } }
I20240502 01:54:34.360184 17498 raft_consensus.cc:2932] T 38a60d7db47149ff90444e5276fc8935 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } } }
I20240502 01:54:34.362825 17350 catalog_manager.cc:5484] T 38a60d7db47149ff90444e5276fc8935 P 1b50a031f2784ba498b3e52e16916205 reported cstate change: config changed from index -1 to 4, NON_VOTER 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) added. New cstate: current_term: 1 leader_uuid: "1b50a031f2784ba498b3e52e16916205" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:34.365869 20744 tablet_bootstrap.cc:492] T df16e178e04040699201f325ce9e2979 P 272b2179c1b4476090a89880df2a8baa: Bootstrap starting.
W20240502 01:54:34.395048 17566 consensus_peers.cc:485] T 38a60d7db47149ff90444e5276fc8935 P 1b50a031f2784ba498b3e52e16916205 -> Peer 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200:37831): Couldn't send request to peer 94931f4b51c9493cb3d4264ce99f5e76. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 38a60d7db47149ff90444e5276fc8935. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:34.395778 19738 raft_consensus.cc:2932] T 768e7cbf63a94d3a815ac77a4d0acf41 P 272b2179c1b4476090a89880df2a8baa [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } } }
I20240502 01:54:34.415513 17767 raft_consensus.cc:2932] T 38a60d7db47149ff90444e5276fc8935 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } } }
I20240502 01:54:34.445315 20881 tablet_bootstrap.cc:492] T 37eda67598fe491bbb29aa62b2c4bec7 P 4b7c382c9edd41babdd1a1afc904a813: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:34.445966 20881 tablet_bootstrap.cc:492] T 37eda67598fe491bbb29aa62b2c4bec7 P 4b7c382c9edd41babdd1a1afc904a813: Bootstrap complete.
I20240502 01:54:34.445569 17632 consensus_queue.cc:237] T 974f67157aa04d3e8b49b264cfa104e8 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } }
I20240502 01:54:34.446517 20881 ts_tablet_manager.cc:1367] T 37eda67598fe491bbb29aa62b2c4bec7 P 4b7c382c9edd41babdd1a1afc904a813: Time spent bootstrapping tablet: real 0.395s	user 0.065s	sys 0.009s
I20240502 01:54:34.459250 20790 tablet_bootstrap.cc:492] T 05b67d9912f047bfb6a2bed90b82ae15 P 4b7c382c9edd41babdd1a1afc904a813: Bootstrap starting.
I20240502 01:54:34.464709 20947 tablet_copy_client.cc:308] T 2c78951695284b1aaab411c0a041c28a P 272b2179c1b4476090a89880df2a8baa: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.194:45617
I20240502 01:54:34.467711 20873 tablet_bootstrap.cc:492] T 37eda67598fe491bbb29aa62b2c4bec7 P 94931f4b51c9493cb3d4264ce99f5e76: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:34.469520 20873 tablet_bootstrap.cc:492] T 37eda67598fe491bbb29aa62b2c4bec7 P 94931f4b51c9493cb3d4264ce99f5e76: Bootstrap complete.
I20240502 01:54:34.470953 20873 ts_tablet_manager.cc:1367] T 37eda67598fe491bbb29aa62b2c4bec7 P 94931f4b51c9493cb3d4264ce99f5e76: Time spent bootstrapping tablet: real 0.401s	user 0.059s	sys 0.013s
I20240502 01:54:34.468562 20783 tablet_bootstrap.cc:492] T 3bdd49939e994240a6d15dbd11559ff4 P 94931f4b51c9493cb3d4264ce99f5e76: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:34.476413 20783 tablet_bootstrap.cc:492] T 3bdd49939e994240a6d15dbd11559ff4 P 94931f4b51c9493cb3d4264ce99f5e76: Bootstrap complete.
I20240502 01:54:34.476914 20783 ts_tablet_manager.cc:1367] T 3bdd49939e994240a6d15dbd11559ff4 P 94931f4b51c9493cb3d4264ce99f5e76: Time spent bootstrapping tablet: real 0.516s	user 0.075s	sys 0.004s
I20240502 01:54:34.487475 20783 raft_consensus.cc:348] T 3bdd49939e994240a6d15dbd11559ff4 P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } }
I20240502 01:54:34.489135 20783 raft_consensus.cc:729] T 3bdd49939e994240a6d15dbd11559ff4 P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 94931f4b51c9493cb3d4264ce99f5e76, State: Initialized, Role: LEARNER
I20240502 01:54:34.488760 20873 raft_consensus.cc:348] T 37eda67598fe491bbb29aa62b2c4bec7 P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } }
I20240502 01:54:34.489990 20873 raft_consensus.cc:729] T 37eda67598fe491bbb29aa62b2c4bec7 P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 94931f4b51c9493cb3d4264ce99f5e76, State: Initialized, Role: LEARNER
I20240502 01:54:34.490886 20873 consensus_queue.cc:260] T 37eda67598fe491bbb29aa62b2c4bec7 P 94931f4b51c9493cb3d4264ce99f5e76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } }
I20240502 01:54:34.490434 20783 consensus_queue.cc:260] T 3bdd49939e994240a6d15dbd11559ff4 P 94931f4b51c9493cb3d4264ce99f5e76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } }
I20240502 01:54:34.494081 17654 tablet_copy_service.cc:140] P 1b50a031f2784ba498b3e52e16916205: Received BeginTabletCopySession request for tablet 2c78951695284b1aaab411c0a041c28a from peer 272b2179c1b4476090a89880df2a8baa ({username='slave'} at 127.9.176.199:56005)
I20240502 01:54:34.496402 17654 tablet_copy_service.cc:161] P 1b50a031f2784ba498b3e52e16916205: Beginning new tablet copy session on tablet 2c78951695284b1aaab411c0a041c28a from peer 272b2179c1b4476090a89880df2a8baa at {username='slave'} at 127.9.176.199:56005: session id = 272b2179c1b4476090a89880df2a8baa-2c78951695284b1aaab411c0a041c28a
I20240502 01:54:34.505734 20783 ts_tablet_manager.cc:1398] T 3bdd49939e994240a6d15dbd11559ff4 P 94931f4b51c9493cb3d4264ce99f5e76: Time spent starting tablet: real 0.029s	user 0.011s	sys 0.000s
I20240502 01:54:34.507269 20873 ts_tablet_manager.cc:1398] T 37eda67598fe491bbb29aa62b2c4bec7 P 94931f4b51c9493cb3d4264ce99f5e76: Time spent starting tablet: real 0.035s	user 0.006s	sys 0.005s
I20240502 01:54:34.507927 17922 tablet_copy_service.cc:342] P da3584f429104d23a1a8899b35cc2138: Request end of tablet copy session 94931f4b51c9493cb3d4264ce99f5e76-3bdd49939e994240a6d15dbd11559ff4 received from {username='slave'} at 127.9.176.200:36935
I20240502 01:54:34.508113 17922 tablet_copy_service.cc:433] P da3584f429104d23a1a8899b35cc2138: ending tablet copy session 94931f4b51c9493cb3d4264ce99f5e76-3bdd49939e994240a6d15dbd11559ff4 on tablet 3bdd49939e994240a6d15dbd11559ff4 with peer 94931f4b51c9493cb3d4264ce99f5e76
I20240502 01:54:34.508582 17520 tablet_copy_service.cc:342] P 311f765f59754bdcb4eadc04e14bd4ea: Request end of tablet copy session 94931f4b51c9493cb3d4264ce99f5e76-37eda67598fe491bbb29aa62b2c4bec7 received from {username='slave'} at 127.9.176.200:46111
I20240502 01:54:34.508728 17520 tablet_copy_service.cc:433] P 311f765f59754bdcb4eadc04e14bd4ea: ending tablet copy session 94931f4b51c9493cb3d4264ce99f5e76-37eda67598fe491bbb29aa62b2c4bec7 on tablet 37eda67598fe491bbb29aa62b2c4bec7 with peer 94931f4b51c9493cb3d4264ce99f5e76
I20240502 01:54:34.510169 17766 raft_consensus.cc:1261] T 974f67157aa04d3e8b49b264cfa104e8 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:34.511845 20628 consensus_queue.cc:1035] T 974f67157aa04d3e8b49b264cfa104e8 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 01:54:34.513604 17565 consensus_peers.cc:485] T 974f67157aa04d3e8b49b264cfa104e8 P 1b50a031f2784ba498b3e52e16916205 -> Peer 29806d3d37354c169484f6db368f8798 (127.9.176.197:35011): Couldn't send request to peer 29806d3d37354c169484f6db368f8798. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 974f67157aa04d3e8b49b264cfa104e8. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:34.516532 17654 tablet_copy_source_session.cc:205] T 2c78951695284b1aaab411c0a041c28a P 1b50a031f2784ba498b3e52e16916205: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:34.518420 17900 raft_consensus.cc:1261] T 974f67157aa04d3e8b49b264cfa104e8 P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:34.518786 20947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c78951695284b1aaab411c0a041c28a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:34.519315 20913 consensus_queue.cc:1035] T 974f67157aa04d3e8b49b264cfa104e8 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:34.528445 20947 tablet_copy_client.cc:783] T 2c78951695284b1aaab411c0a041c28a P 272b2179c1b4476090a89880df2a8baa: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:34.528803 20947 tablet_copy_client.cc:647] T 2c78951695284b1aaab411c0a041c28a P 272b2179c1b4476090a89880df2a8baa: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:34.529485 20913 raft_consensus.cc:2932] T 974f67157aa04d3e8b49b264cfa104e8 P 1b50a031f2784ba498b3e52e16916205 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 29806d3d37354c169484f6db368f8798 (127.9.176.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } } }
I20240502 01:54:34.531739 20947 tablet_copy_client.cc:523] T 2c78951695284b1aaab411c0a041c28a P 272b2179c1b4476090a89880df2a8baa: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:34.532701 17767 raft_consensus.cc:2932] T 974f67157aa04d3e8b49b264cfa104e8 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 29806d3d37354c169484f6db368f8798 (127.9.176.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } } }
I20240502 01:54:34.533463 20881 raft_consensus.cc:348] T 37eda67598fe491bbb29aa62b2c4bec7 P 4b7c382c9edd41babdd1a1afc904a813 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } }
I20240502 01:54:34.535319 20881 raft_consensus.cc:729] T 37eda67598fe491bbb29aa62b2c4bec7 P 4b7c382c9edd41babdd1a1afc904a813 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4b7c382c9edd41babdd1a1afc904a813, State: Initialized, Role: LEARNER
I20240502 01:54:34.542337 17350 catalog_manager.cc:5484] T 974f67157aa04d3e8b49b264cfa104e8 P 1b50a031f2784ba498b3e52e16916205 reported cstate change: config changed from index -1 to 4, NON_VOTER 29806d3d37354c169484f6db368f8798 (127.9.176.197) added. New cstate: current_term: 1 leader_uuid: "1b50a031f2784ba498b3e52e16916205" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:34.543304 20947 tablet_bootstrap.cc:492] T 2c78951695284b1aaab411c0a041c28a P 272b2179c1b4476090a89880df2a8baa: Bootstrap starting.
I20240502 01:54:34.539824 20881 consensus_queue.cc:260] T 37eda67598fe491bbb29aa62b2c4bec7 P 4b7c382c9edd41babdd1a1afc904a813 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } }
I20240502 01:54:34.560217 20920 raft_consensus.cc:1050] T 3b3fa060e8d64911bec810ec07a358d2 P 1b50a031f2784ba498b3e52e16916205: attempting to promote NON_VOTER 272b2179c1b4476090a89880df2a8baa to VOTER
I20240502 01:54:34.563596 20920 consensus_queue.cc:237] T 3b3fa060e8d64911bec810ec07a358d2 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } }
I20240502 01:54:34.567116 17898 raft_consensus.cc:2932] T 974f67157aa04d3e8b49b264cfa104e8 P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 29806d3d37354c169484f6db368f8798 (127.9.176.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } } }
I20240502 01:54:34.568679 20881 ts_tablet_manager.cc:1398] T 37eda67598fe491bbb29aa62b2c4bec7 P 4b7c382c9edd41babdd1a1afc904a813: Time spent starting tablet: real 0.122s	user 0.009s	sys 0.006s
I20240502 01:54:34.573549 19871 raft_consensus.cc:1203] T 37eda67598fe491bbb29aa62b2c4bec7 P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
I20240502 01:54:34.578739 17520 tablet_copy_service.cc:342] P 311f765f59754bdcb4eadc04e14bd4ea: Request end of tablet copy session 4b7c382c9edd41babdd1a1afc904a813-37eda67598fe491bbb29aa62b2c4bec7 received from {username='slave'} at 127.9.176.198:53399
I20240502 01:54:34.578920 17520 tablet_copy_service.cc:433] P 311f765f59754bdcb4eadc04e14bd4ea: ending tablet copy session 4b7c382c9edd41babdd1a1afc904a813-37eda67598fe491bbb29aa62b2c4bec7 on tablet 37eda67598fe491bbb29aa62b2c4bec7 with peer 4b7c382c9edd41babdd1a1afc904a813
I20240502 01:54:34.581043 17900 raft_consensus.cc:1261] T 3b3fa060e8d64911bec810ec07a358d2 P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:34.581979 19871 raft_consensus.cc:1203] T 3bdd49939e994240a6d15dbd11559ff4 P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 01:54:34.584450 20913 consensus_queue.cc:1035] T 3b3fa060e8d64911bec810ec07a358d2 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:34.589305 19738 raft_consensus.cc:1261] T 3b3fa060e8d64911bec810ec07a358d2 P 272b2179c1b4476090a89880df2a8baa [term 1 LEARNER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:34.592749 17498 raft_consensus.cc:1261] T 3b3fa060e8d64911bec810ec07a358d2 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:34.594264 20913 consensus_queue.cc:1035] T 3b3fa060e8d64911bec810ec07a358d2 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:34.601115 20913 consensus_queue.cc:1035] T 3b3fa060e8d64911bec810ec07a358d2 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.002s
I20240502 01:54:34.606704 20628 raft_consensus.cc:2932] T 3b3fa060e8d64911bec810ec07a358d2 P 1b50a031f2784ba498b3e52e16916205 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 272b2179c1b4476090a89880df2a8baa (127.9.176.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } }
I20240502 01:54:34.614550 19467 raft_consensus.cc:1203] T 6da59c0626c64a5988e39a831c42c268 P 29806d3d37354c169484f6db368f8798 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
I20240502 01:54:34.614069 19738 raft_consensus.cc:2932] T 3b3fa060e8d64911bec810ec07a358d2 P 272b2179c1b4476090a89880df2a8baa [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 272b2179c1b4476090a89880df2a8baa (127.9.176.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } }
I20240502 01:54:34.617444 17498 raft_consensus.cc:2932] T 3b3fa060e8d64911bec810ec07a358d2 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 272b2179c1b4476090a89880df2a8baa (127.9.176.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } }
I20240502 01:54:34.625718 17632 consensus_queue.cc:237] T b40f968655774ce887dff8b7f4651501 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } }
I20240502 01:54:34.629837 17355 catalog_manager.cc:5484] T 3b3fa060e8d64911bec810ec07a358d2 P 1b50a031f2784ba498b3e52e16916205 reported cstate change: config changed from index 4 to 5, 272b2179c1b4476090a89880df2a8baa (127.9.176.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1b50a031f2784ba498b3e52e16916205" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:34.646548 17634 consensus_queue.cc:237] T 3b3fa060e8d64911bec810ec07a358d2 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } }
I20240502 01:54:34.650009 17900 raft_consensus.cc:1261] T b40f968655774ce887dff8b7f4651501 P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:34.650687 20635 consensus_queue.cc:1035] T b40f968655774ce887dff8b7f4651501 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:34.651082 17498 raft_consensus.cc:1261] T 3b3fa060e8d64911bec810ec07a358d2 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:34.653023 20635 consensus_queue.cc:1035] T 3b3fa060e8d64911bec810ec07a358d2 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240502 01:54:34.655803 17565 consensus_peers.cc:485] T b40f968655774ce887dff8b7f4651501 P 1b50a031f2784ba498b3e52e16916205 -> Peer 272b2179c1b4476090a89880df2a8baa (127.9.176.199:42269): Couldn't send request to peer 272b2179c1b4476090a89880df2a8baa. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:34.658105 20635 raft_consensus.cc:2932] T 3b3fa060e8d64911bec810ec07a358d2 P 1b50a031f2784ba498b3e52e16916205 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER da3584f429104d23a1a8899b35cc2138 (127.9.176.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } }
I20240502 01:54:34.662858 17343 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3b3fa060e8d64911bec810ec07a358d2 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:34.664446 19736 raft_consensus.cc:1261] T 3b3fa060e8d64911bec810ec07a358d2 P 272b2179c1b4476090a89880df2a8baa [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:34.664265 17498 raft_consensus.cc:2932] T 3b3fa060e8d64911bec810ec07a358d2 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER da3584f429104d23a1a8899b35cc2138 (127.9.176.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } }
I20240502 01:54:34.669440 20920 consensus_queue.cc:1035] T 3b3fa060e8d64911bec810ec07a358d2 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:34.672653 17348 catalog_manager.cc:5484] T 3b3fa060e8d64911bec810ec07a358d2 P 311f765f59754bdcb4eadc04e14bd4ea reported cstate change: config changed from index 5 to 6, VOTER da3584f429104d23a1a8899b35cc2138 (127.9.176.196) evicted. New cstate: current_term: 1 leader_uuid: "1b50a031f2784ba498b3e52e16916205" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } }
I20240502 01:54:34.676431 17766 raft_consensus.cc:1261] T b40f968655774ce887dff8b7f4651501 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:34.678260 17880 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b3fa060e8d64911bec810ec07a358d2 with delete_type TABLET_DATA_TOMBSTONED (TS da3584f429104d23a1a8899b35cc2138 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:34396
I20240502 01:54:34.687387 20635 consensus_queue.cc:1035] T b40f968655774ce887dff8b7f4651501 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:34.687388 19736 raft_consensus.cc:2932] T 3b3fa060e8d64911bec810ec07a358d2 P 272b2179c1b4476090a89880df2a8baa [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER da3584f429104d23a1a8899b35cc2138 (127.9.176.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } }
I20240502 01:54:34.703774 20911 raft_consensus.cc:2932] T b40f968655774ce887dff8b7f4651501 P 1b50a031f2784ba498b3e52e16916205 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } } }
I20240502 01:54:34.704767 17766 raft_consensus.cc:2932] T b40f968655774ce887dff8b7f4651501 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } } }
I20240502 01:54:34.708685 17898 raft_consensus.cc:2932] T b40f968655774ce887dff8b7f4651501 P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } } }
I20240502 01:54:34.711709 17350 catalog_manager.cc:5484] T b40f968655774ce887dff8b7f4651501 P 84d3a4b174c34e02aedab1e620731359 reported cstate change: config changed from index 4 to 5, NON_VOTER 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) added. New cstate: current_term: 1 leader_uuid: "1b50a031f2784ba498b3e52e16916205" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } } }
I20240502 01:54:34.723014 20619 raft_consensus.cc:1050] T c91f74f46b04425aa5393515cb9f781c P 84d3a4b174c34e02aedab1e620731359: attempting to promote NON_VOTER 29806d3d37354c169484f6db368f8798 to VOTER
I20240502 01:54:34.724154 20619 consensus_queue.cc:237] T c91f74f46b04425aa5393515cb9f781c P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } }
I20240502 01:54:34.735810 19467 raft_consensus.cc:1261] T c91f74f46b04425aa5393515cb9f781c P 29806d3d37354c169484f6db368f8798 [term 1 LEARNER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:34.735965 17497 raft_consensus.cc:1261] T c91f74f46b04425aa5393515cb9f781c P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:34.736625 20618 consensus_queue.cc:1035] T c91f74f46b04425aa5393515cb9f781c P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:34.737258 20877 tablet_bootstrap.cc:492] T 2045e08a200148c69326351f38fd484d P 29806d3d37354c169484f6db368f8798: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:34.737303 17900 raft_consensus.cc:1261] T c91f74f46b04425aa5393515cb9f781c P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:34.738564 20618 consensus_queue.cc:1035] T c91f74f46b04425aa5393515cb9f781c P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:34.739320 20619 consensus_queue.cc:1035] T c91f74f46b04425aa5393515cb9f781c P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:34.743379 20877 tablet_bootstrap.cc:492] T 2045e08a200148c69326351f38fd484d P 29806d3d37354c169484f6db368f8798: Bootstrap complete.
I20240502 01:54:34.748039 20877 ts_tablet_manager.cc:1367] T 2045e08a200148c69326351f38fd484d P 29806d3d37354c169484f6db368f8798: Time spent bootstrapping tablet: real 0.526s	user 0.061s	sys 0.011s
I20240502 01:54:34.755769 20996 tablet_replica.cc:330] T 3b3fa060e8d64911bec810ec07a358d2 P da3584f429104d23a1a8899b35cc2138: stopping tablet replica
I20240502 01:54:34.756321 20996 raft_consensus.cc:2229] T 3b3fa060e8d64911bec810ec07a358d2 P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:34.756642 20996 pending_rounds.cc:62] T 3b3fa060e8d64911bec810ec07a358d2 P da3584f429104d23a1a8899b35cc2138: Trying to abort 1 pending ops.
I20240502 01:54:34.756763 20996 pending_rounds.cc:68] T 3b3fa060e8d64911bec810ec07a358d2 P da3584f429104d23a1a8899b35cc2138: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7023062526201372672 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "3b3fa060e8d64911bec810ec07a358d2" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } } }
I20240502 01:54:34.757242 20996 raft_consensus.cc:2894] T 3b3fa060e8d64911bec810ec07a358d2 P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240502 01:54:34.757382 20996 raft_consensus.cc:2866] T 3b3fa060e8d64911bec810ec07a358d2 P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240502 01:54:34.757514 20996 raft_consensus.cc:2258] T 3b3fa060e8d64911bec810ec07a358d2 P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:34.755661 20877 raft_consensus.cc:348] T 2045e08a200148c69326351f38fd484d P 29806d3d37354c169484f6db368f8798 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } }
I20240502 01:54:34.764142 20877 raft_consensus.cc:729] T 2045e08a200148c69326351f38fd484d P 29806d3d37354c169484f6db368f8798 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 29806d3d37354c169484f6db368f8798, State: Initialized, Role: LEARNER
I20240502 01:54:34.764628 20877 consensus_queue.cc:260] T 2045e08a200148c69326351f38fd484d P 29806d3d37354c169484f6db368f8798 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } }
I20240502 01:54:34.772713 20293 raft_consensus.cc:2932] T c91f74f46b04425aa5393515cb9f781c P 84d3a4b174c34e02aedab1e620731359 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 29806d3d37354c169484f6db368f8798 (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } }
I20240502 01:54:34.781877 19604 raft_consensus.cc:1203] T 37eda67598fe491bbb29aa62b2c4bec7 P 4b7c382c9edd41babdd1a1afc904a813 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
I20240502 01:54:34.793939 17497 raft_consensus.cc:2932] T c91f74f46b04425aa5393515cb9f781c P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 29806d3d37354c169484f6db368f8798 (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } }
W20240502 01:54:34.801436 17565 consensus_peers.cc:485] T b40f968655774ce887dff8b7f4651501 P 1b50a031f2784ba498b3e52e16916205 -> Peer 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198:36865): Couldn't send request to peer 4b7c382c9edd41babdd1a1afc904a813. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b40f968655774ce887dff8b7f4651501. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:34.801465 20877 ts_tablet_manager.cc:1398] T 2045e08a200148c69326351f38fd484d P 29806d3d37354c169484f6db368f8798: Time spent starting tablet: real 0.048s	user 0.003s	sys 0.000s
I20240502 01:54:34.805148 20628 raft_consensus.cc:1050] T 91e0e82526c0489a8908e250bf3c851f P 1b50a031f2784ba498b3e52e16916205: attempting to promote NON_VOTER 29806d3d37354c169484f6db368f8798 to VOTER
I20240502 01:54:34.806020 20628 consensus_queue.cc:237] T 91e0e82526c0489a8908e250bf3c851f P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } }
I20240502 01:54:34.820492 20783 ts_tablet_manager.cc:902] T 38a60d7db47149ff90444e5276fc8935 P 94931f4b51c9493cb3d4264ce99f5e76: Initiating tablet copy from peer 1b50a031f2784ba498b3e52e16916205 (127.9.176.194:45617)
I20240502 01:54:34.838251 17350 catalog_manager.cc:5484] T c91f74f46b04425aa5393515cb9f781c P 311f765f59754bdcb4eadc04e14bd4ea reported cstate change: config changed from index 4 to 5, 29806d3d37354c169484f6db368f8798 (127.9.176.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "84d3a4b174c34e02aedab1e620731359" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } }
I20240502 01:54:34.843009 17766 consensus_queue.cc:237] T 88a137bcfc76410583f910b377d1be7f P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } }
I20240502 01:54:34.858134 20744 tablet_bootstrap.cc:492] T df16e178e04040699201f325ce9e2979 P 272b2179c1b4476090a89880df2a8baa: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:34.858661 20744 tablet_bootstrap.cc:492] T df16e178e04040699201f325ce9e2979 P 272b2179c1b4476090a89880df2a8baa: Bootstrap complete.
I20240502 01:54:34.858954 20744 ts_tablet_manager.cc:1367] T df16e178e04040699201f325ce9e2979 P 272b2179c1b4476090a89880df2a8baa: Time spent bootstrapping tablet: real 0.493s	user 0.050s	sys 0.003s
I20240502 01:54:34.859865 20744 raft_consensus.cc:348] T df16e178e04040699201f325ce9e2979 P 272b2179c1b4476090a89880df2a8baa [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } }
I20240502 01:54:34.860139 20744 raft_consensus.cc:729] T df16e178e04040699201f325ce9e2979 P 272b2179c1b4476090a89880df2a8baa [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 272b2179c1b4476090a89880df2a8baa, State: Initialized, Role: LEARNER
I20240502 01:54:34.860368 20744 consensus_queue.cc:260] T df16e178e04040699201f325ce9e2979 P 272b2179c1b4476090a89880df2a8baa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } }
I20240502 01:54:34.870900 20744 ts_tablet_manager.cc:1398] T df16e178e04040699201f325ce9e2979 P 272b2179c1b4476090a89880df2a8baa: Time spent starting tablet: real 0.012s	user 0.005s	sys 0.000s
I20240502 01:54:34.871721 20699 tablet_bootstrap.cc:492] T b40f968655774ce887dff8b7f4651501 P 272b2179c1b4476090a89880df2a8baa: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:34.871884 17654 tablet_copy_service.cc:342] P 1b50a031f2784ba498b3e52e16916205: Request end of tablet copy session 272b2179c1b4476090a89880df2a8baa-df16e178e04040699201f325ce9e2979 received from {username='slave'} at 127.9.176.199:56005
I20240502 01:54:34.872058 17654 tablet_copy_service.cc:433] P 1b50a031f2784ba498b3e52e16916205: ending tablet copy session 272b2179c1b4476090a89880df2a8baa-df16e178e04040699201f325ce9e2979 on tablet df16e178e04040699201f325ce9e2979 with peer 272b2179c1b4476090a89880df2a8baa
I20240502 01:54:34.872195 20699 tablet_bootstrap.cc:492] T b40f968655774ce887dff8b7f4651501 P 272b2179c1b4476090a89880df2a8baa: Bootstrap complete.
I20240502 01:54:34.872457 20699 ts_tablet_manager.cc:1367] T b40f968655774ce887dff8b7f4651501 P 272b2179c1b4476090a89880df2a8baa: Time spent bootstrapping tablet: real 0.735s	user 0.049s	sys 0.010s
I20240502 01:54:34.873288 20699 raft_consensus.cc:348] T b40f968655774ce887dff8b7f4651501 P 272b2179c1b4476090a89880df2a8baa [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } }
I20240502 01:54:34.873584 20699 raft_consensus.cc:729] T b40f968655774ce887dff8b7f4651501 P 272b2179c1b4476090a89880df2a8baa [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 272b2179c1b4476090a89880df2a8baa, State: Initialized, Role: LEARNER
I20240502 01:54:34.874212 17900 raft_consensus.cc:2932] T c91f74f46b04425aa5393515cb9f781c P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 29806d3d37354c169484f6db368f8798 (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } }
I20240502 01:54:34.880153 19467 raft_consensus.cc:1261] T 91e0e82526c0489a8908e250bf3c851f P 29806d3d37354c169484f6db368f8798 [term 1 LEARNER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:34.881001 20635 consensus_queue.cc:1035] T 91e0e82526c0489a8908e250bf3c851f P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:34.884938 20699 consensus_queue.cc:260] T b40f968655774ce887dff8b7f4651501 P 272b2179c1b4476090a89880df2a8baa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } }
I20240502 01:54:34.886260 20699 ts_tablet_manager.cc:1398] T b40f968655774ce887dff8b7f4651501 P 272b2179c1b4476090a89880df2a8baa: Time spent starting tablet: real 0.014s	user 0.003s	sys 0.000s
I20240502 01:54:34.887075 17654 tablet_copy_service.cc:342] P 1b50a031f2784ba498b3e52e16916205: Request end of tablet copy session 272b2179c1b4476090a89880df2a8baa-b40f968655774ce887dff8b7f4651501 received from {username='slave'} at 127.9.176.199:56005
I20240502 01:54:34.887245 17654 tablet_copy_service.cc:433] P 1b50a031f2784ba498b3e52e16916205: ending tablet copy session 272b2179c1b4476090a89880df2a8baa-b40f968655774ce887dff8b7f4651501 on tablet b40f968655774ce887dff8b7f4651501 with peer 272b2179c1b4476090a89880df2a8baa
I20240502 01:54:34.898931 17900 raft_consensus.cc:1261] T 91e0e82526c0489a8908e250bf3c851f P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:34.899931 20920 consensus_queue.cc:1035] T 91e0e82526c0489a8908e250bf3c851f P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:34.897955 20790 tablet_bootstrap.cc:492] T 05b67d9912f047bfb6a2bed90b82ae15 P 4b7c382c9edd41babdd1a1afc904a813: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:34.902244 20790 tablet_bootstrap.cc:492] T 05b67d9912f047bfb6a2bed90b82ae15 P 4b7c382c9edd41babdd1a1afc904a813: Bootstrap complete.
I20240502 01:54:34.902629 20790 ts_tablet_manager.cc:1367] T 05b67d9912f047bfb6a2bed90b82ae15 P 4b7c382c9edd41babdd1a1afc904a813: Time spent bootstrapping tablet: real 0.608s	user 0.051s	sys 0.013s
I20240502 01:54:34.903821 20790 raft_consensus.cc:348] T 05b67d9912f047bfb6a2bed90b82ae15 P 4b7c382c9edd41babdd1a1afc904a813 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } }
I20240502 01:54:34.904181 20790 raft_consensus.cc:729] T 05b67d9912f047bfb6a2bed90b82ae15 P 4b7c382c9edd41babdd1a1afc904a813 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4b7c382c9edd41babdd1a1afc904a813, State: Initialized, Role: LEARNER
I20240502 01:54:34.904431 20790 consensus_queue.cc:260] T 05b67d9912f047bfb6a2bed90b82ae15 P 4b7c382c9edd41babdd1a1afc904a813 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } }
I20240502 01:54:34.905205 20790 ts_tablet_manager.cc:1398] T 05b67d9912f047bfb6a2bed90b82ae15 P 4b7c382c9edd41babdd1a1afc904a813: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 01:54:34.906221 17922 tablet_copy_service.cc:342] P da3584f429104d23a1a8899b35cc2138: Request end of tablet copy session 4b7c382c9edd41babdd1a1afc904a813-05b67d9912f047bfb6a2bed90b82ae15 received from {username='slave'} at 127.9.176.198:45579
I20240502 01:54:34.906363 17922 tablet_copy_service.cc:433] P da3584f429104d23a1a8899b35cc2138: ending tablet copy session 4b7c382c9edd41babdd1a1afc904a813-05b67d9912f047bfb6a2bed90b82ae15 on tablet 05b67d9912f047bfb6a2bed90b82ae15 with peer 4b7c382c9edd41babdd1a1afc904a813
I20240502 01:54:34.908331 20947 tablet_bootstrap.cc:492] T 2c78951695284b1aaab411c0a041c28a P 272b2179c1b4476090a89880df2a8baa: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:34.908872 20947 tablet_bootstrap.cc:492] T 2c78951695284b1aaab411c0a041c28a P 272b2179c1b4476090a89880df2a8baa: Bootstrap complete.
I20240502 01:54:34.909307 20947 ts_tablet_manager.cc:1367] T 2c78951695284b1aaab411c0a041c28a P 272b2179c1b4476090a89880df2a8baa: Time spent bootstrapping tablet: real 0.366s	user 0.060s	sys 0.000s
I20240502 01:54:34.910522 20947 raft_consensus.cc:348] T 2c78951695284b1aaab411c0a041c28a P 272b2179c1b4476090a89880df2a8baa [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } }
I20240502 01:54:34.910851 20947 raft_consensus.cc:729] T 2c78951695284b1aaab411c0a041c28a P 272b2179c1b4476090a89880df2a8baa [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 272b2179c1b4476090a89880df2a8baa, State: Initialized, Role: LEARNER
I20240502 01:54:34.911155 20947 consensus_queue.cc:260] T 2c78951695284b1aaab411c0a041c28a P 272b2179c1b4476090a89880df2a8baa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } }
I20240502 01:54:34.911960 20947 ts_tablet_manager.cc:1398] T 2c78951695284b1aaab411c0a041c28a P 272b2179c1b4476090a89880df2a8baa: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 01:54:34.912958 17654 tablet_copy_service.cc:342] P 1b50a031f2784ba498b3e52e16916205: Request end of tablet copy session 272b2179c1b4476090a89880df2a8baa-2c78951695284b1aaab411c0a041c28a received from {username='slave'} at 127.9.176.199:56005
I20240502 01:54:34.913166 17654 tablet_copy_service.cc:433] P 1b50a031f2784ba498b3e52e16916205: ending tablet copy session 272b2179c1b4476090a89880df2a8baa-2c78951695284b1aaab411c0a041c28a on tablet 2c78951695284b1aaab411c0a041c28a with peer 272b2179c1b4476090a89880df2a8baa
I20240502 01:54:34.917446 17497 raft_consensus.cc:1261] T 91e0e82526c0489a8908e250bf3c851f P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:34.918241 20920 consensus_queue.cc:1035] T 91e0e82526c0489a8908e250bf3c851f P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:34.934245 20878 tablet_bootstrap.cc:492] T 768e7cbf63a94d3a815ac77a4d0acf41 P 4b7c382c9edd41babdd1a1afc904a813: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:34.934784 20878 tablet_bootstrap.cc:492] T 768e7cbf63a94d3a815ac77a4d0acf41 P 4b7c382c9edd41babdd1a1afc904a813: Bootstrap complete.
I20240502 01:54:34.935097 20878 ts_tablet_manager.cc:1367] T 768e7cbf63a94d3a815ac77a4d0acf41 P 4b7c382c9edd41babdd1a1afc904a813: Time spent bootstrapping tablet: real 0.717s	user 0.070s	sys 0.029s
I20240502 01:54:34.935950 20878 raft_consensus.cc:348] T 768e7cbf63a94d3a815ac77a4d0acf41 P 4b7c382c9edd41babdd1a1afc904a813 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } }
I20240502 01:54:34.936226 20878 raft_consensus.cc:729] T 768e7cbf63a94d3a815ac77a4d0acf41 P 4b7c382c9edd41babdd1a1afc904a813 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4b7c382c9edd41babdd1a1afc904a813, State: Initialized, Role: LEARNER
I20240502 01:54:34.936424 20878 consensus_queue.cc:260] T 768e7cbf63a94d3a815ac77a4d0acf41 P 4b7c382c9edd41babdd1a1afc904a813 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } }
I20240502 01:54:34.938083 20878 ts_tablet_manager.cc:1398] T 768e7cbf63a94d3a815ac77a4d0acf41 P 4b7c382c9edd41babdd1a1afc904a813: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.003s
I20240502 01:54:34.942277 17789 tablet_copy_service.cc:342] P 84d3a4b174c34e02aedab1e620731359: Request end of tablet copy session 29806d3d37354c169484f6db368f8798-2045e08a200148c69326351f38fd484d received from {username='slave'} at 127.9.176.197:36787
I20240502 01:54:34.942441 17789 tablet_copy_service.cc:433] P 84d3a4b174c34e02aedab1e620731359: ending tablet copy session 29806d3d37354c169484f6db368f8798-2045e08a200148c69326351f38fd484d on tablet 2045e08a200148c69326351f38fd484d with peer 29806d3d37354c169484f6db368f8798
I20240502 01:54:34.949620 17765 consensus_queue.cc:237] T c91f74f46b04425aa5393515cb9f781c P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } }
I20240502 01:54:34.953842 20877 ts_tablet_manager.cc:902] T 974f67157aa04d3e8b49b264cfa104e8 P 29806d3d37354c169484f6db368f8798: Initiating tablet copy from peer 1b50a031f2784ba498b3e52e16916205 (127.9.176.194:45617)
I20240502 01:54:34.961475 17499 raft_consensus.cc:1261] T 88a137bcfc76410583f910b377d1be7f P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:34.962718 21006 consensus_queue.cc:1035] T 88a137bcfc76410583f910b377d1be7f P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:34.964040 20635 raft_consensus.cc:2932] T 91e0e82526c0489a8908e250bf3c851f P 1b50a031f2784ba498b3e52e16916205 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 29806d3d37354c169484f6db368f8798 (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } }
I20240502 01:54:34.965291 20879 tablet_bootstrap.cc:492] T 6da59c0626c64a5988e39a831c42c268 P 94931f4b51c9493cb3d4264ce99f5e76: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:34.965744 20879 tablet_bootstrap.cc:492] T 6da59c0626c64a5988e39a831c42c268 P 94931f4b51c9493cb3d4264ce99f5e76: Bootstrap complete.
I20240502 01:54:34.966025 20879 ts_tablet_manager.cc:1367] T 6da59c0626c64a5988e39a831c42c268 P 94931f4b51c9493cb3d4264ce99f5e76: Time spent bootstrapping tablet: real 0.794s	user 0.051s	sys 0.053s
I20240502 01:54:34.966886 20879 raft_consensus.cc:348] T 6da59c0626c64a5988e39a831c42c268 P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } }
I20240502 01:54:34.967180 20879 raft_consensus.cc:729] T 6da59c0626c64a5988e39a831c42c268 P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 94931f4b51c9493cb3d4264ce99f5e76, State: Initialized, Role: LEARNER
I20240502 01:54:34.967386 20879 consensus_queue.cc:260] T 6da59c0626c64a5988e39a831c42c268 P 94931f4b51c9493cb3d4264ce99f5e76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } }
I20240502 01:54:34.968120 20879 ts_tablet_manager.cc:1398] T 6da59c0626c64a5988e39a831c42c268 P 94931f4b51c9493cb3d4264ce99f5e76: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.000s
I20240502 01:54:34.981678 17349 catalog_manager.cc:5484] T 91e0e82526c0489a8908e250bf3c851f P 1b50a031f2784ba498b3e52e16916205 reported cstate change: config changed from index 4 to 5, 29806d3d37354c169484f6db368f8798 (127.9.176.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1b50a031f2784ba498b3e52e16916205" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:34.993407 19467 raft_consensus.cc:2932] T 91e0e82526c0489a8908e250bf3c851f P 29806d3d37354c169484f6db368f8798 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 29806d3d37354c169484f6db368f8798 (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } }
I20240502 01:54:35.000880 17497 raft_consensus.cc:2932] T 91e0e82526c0489a8908e250bf3c851f P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 29806d3d37354c169484f6db368f8798 (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } }
I20240502 01:54:35.002138 17900 raft_consensus.cc:2932] T 91e0e82526c0489a8908e250bf3c851f P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 29806d3d37354c169484f6db368f8798 (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } }
I20240502 01:54:35.008096 20937 raft_consensus.cc:1050] T 3bdd49939e994240a6d15dbd11559ff4 P da3584f429104d23a1a8899b35cc2138: attempting to promote NON_VOTER 94931f4b51c9493cb3d4264ce99f5e76 to VOTER
I20240502 01:54:35.009600 20937 consensus_queue.cc:237] T 3bdd49939e994240a6d15dbd11559ff4 P da3584f429104d23a1a8899b35cc2138 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } }
I20240502 01:54:35.013909 17634 raft_consensus.cc:1261] T 88a137bcfc76410583f910b377d1be7f P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:35.014737 20293 consensus_queue.cc:1035] T 88a137bcfc76410583f910b377d1be7f P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:35.017841 17632 consensus_queue.cc:237] T 91e0e82526c0489a8908e250bf3c851f P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } }
I20240502 01:54:35.019898 19736 raft_consensus.cc:1203] T df16e178e04040699201f325ce9e2979 P 272b2179c1b4476090a89880df2a8baa [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 01:54:35.020990 17497 raft_consensus.cc:1261] T 91e0e82526c0489a8908e250bf3c851f P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:35.021728 20635 consensus_queue.cc:1035] T 91e0e82526c0489a8908e250bf3c851f P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:35.025030 19871 raft_consensus.cc:1261] T 3bdd49939e994240a6d15dbd11559ff4 P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 LEARNER]: Refusing update from remote peer da3584f429104d23a1a8899b35cc2138: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:35.026415 20915 consensus_queue.cc:1035] T 3bdd49939e994240a6d15dbd11559ff4 P da3584f429104d23a1a8899b35cc2138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:35.027875 19736 raft_consensus.cc:2932] T b40f968655774ce887dff8b7f4651501 P 272b2179c1b4476090a89880df2a8baa [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } } }
I20240502 01:54:35.035604 20913 raft_consensus.cc:2932] T 91e0e82526c0489a8908e250bf3c851f P 1b50a031f2784ba498b3e52e16916205 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER da3584f429104d23a1a8899b35cc2138 (127.9.176.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } }
I20240502 01:54:35.041591 17765 raft_consensus.cc:1261] T 3bdd49939e994240a6d15dbd11559ff4 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Refusing update from remote peer da3584f429104d23a1a8899b35cc2138: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:35.042908 20915 consensus_queue.cc:1035] T 3bdd49939e994240a6d15dbd11559ff4 P da3584f429104d23a1a8899b35cc2138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:35.044150 17349 catalog_manager.cc:5484] T 91e0e82526c0489a8908e250bf3c851f P 1b50a031f2784ba498b3e52e16916205 reported cstate change: config changed from index 5 to 6, VOTER da3584f429104d23a1a8899b35cc2138 (127.9.176.196) evicted. New cstate: current_term: 1 leader_uuid: "1b50a031f2784ba498b3e52e16916205" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:35.046000 20293 raft_consensus.cc:2932] T 88a137bcfc76410583f910b377d1be7f P 84d3a4b174c34e02aedab1e620731359 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } } }
I20240502 01:54:35.049760 17498 raft_consensus.cc:2932] T 91e0e82526c0489a8908e250bf3c851f P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER da3584f429104d23a1a8899b35cc2138 (127.9.176.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } }
I20240502 01:54:35.052703 20788 raft_consensus.cc:1050] T 37eda67598fe491bbb29aa62b2c4bec7 P 311f765f59754bdcb4eadc04e14bd4ea: attempting to promote NON_VOTER 94931f4b51c9493cb3d4264ce99f5e76 to VOTER
I20240502 01:54:35.054070 20788 consensus_queue.cc:237] T 37eda67598fe491bbb29aa62b2c4bec7 P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } }
I20240502 01:54:35.054728 17632 raft_consensus.cc:1261] T 3bdd49939e994240a6d15dbd11559ff4 P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Refusing update from remote peer da3584f429104d23a1a8899b35cc2138: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:35.055474 20937 consensus_queue.cc:1035] T 3bdd49939e994240a6d15dbd11559ff4 P da3584f429104d23a1a8899b35cc2138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:35.057350 17351 catalog_manager.cc:5484] T 88a137bcfc76410583f910b377d1be7f P 84d3a4b174c34e02aedab1e620731359 reported cstate change: config changed from index -1 to 4, NON_VOTER 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) added. New cstate: current_term: 1 leader_uuid: "84d3a4b174c34e02aedab1e620731359" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:35.064236 17343 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 91e0e82526c0489a8908e250bf3c851f with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:35.071856 17880 tablet_service.cc:1514] Processing DeleteTablet for tablet 91e0e82526c0489a8908e250bf3c851f with delete_type TABLET_DATA_TOMBSTONED (TS da3584f429104d23a1a8899b35cc2138 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:34396
I20240502 01:54:35.075244 17498 raft_consensus.cc:2932] T 88a137bcfc76410583f910b377d1be7f P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } } }
I20240502 01:54:35.076817 17632 raft_consensus.cc:2932] T 88a137bcfc76410583f910b377d1be7f P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } } }
I20240502 01:54:35.074597 20915 raft_consensus.cc:2932] T 3bdd49939e994240a6d15dbd11559ff4 P da3584f429104d23a1a8899b35cc2138 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
I20240502 01:54:35.083597 17789 tablet_copy_service.cc:342] P 84d3a4b174c34e02aedab1e620731359: Request end of tablet copy session 4b7c382c9edd41babdd1a1afc904a813-768e7cbf63a94d3a815ac77a4d0acf41 received from {username='slave'} at 127.9.176.198:55177
I20240502 01:54:35.083742 17789 tablet_copy_service.cc:433] P 84d3a4b174c34e02aedab1e620731359: ending tablet copy session 4b7c382c9edd41babdd1a1afc904a813-768e7cbf63a94d3a815ac77a4d0acf41 on tablet 768e7cbf63a94d3a815ac77a4d0acf41 with peer 4b7c382c9edd41babdd1a1afc904a813
I20240502 01:54:35.085142 20619 raft_consensus.cc:1050] T 768e7cbf63a94d3a815ac77a4d0acf41 P 84d3a4b174c34e02aedab1e620731359: attempting to promote NON_VOTER 272b2179c1b4476090a89880df2a8baa to VOTER
I20240502 01:54:35.086222 20619 consensus_queue.cc:237] T 768e7cbf63a94d3a815ac77a4d0acf41 P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } }
I20240502 01:54:35.083460 19465 raft_consensus.cc:1203] T 2045e08a200148c69326351f38fd484d P 29806d3d37354c169484f6db368f8798 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 01:54:35.093483 19871 raft_consensus.cc:2932] T 3bdd49939e994240a6d15dbd11559ff4 P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
I20240502 01:54:35.095466 20996 ts_tablet_manager.cc:1875] T 3b3fa060e8d64911bec810ec07a358d2 P da3584f429104d23a1a8899b35cc2138: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:35.100471 17765 raft_consensus.cc:2932] T 3bdd49939e994240a6d15dbd11559ff4 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
I20240502 01:54:35.105593 17898 consensus_queue.cc:237] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P da3584f429104d23a1a8899b35cc2138 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } }
I20240502 01:54:35.106205 17498 raft_consensus.cc:1261] T c91f74f46b04425aa5393515cb9f781c P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:35.107180 20293 consensus_queue.cc:1035] T c91f74f46b04425aa5393515cb9f781c P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:35.107286 17634 raft_consensus.cc:2932] T 3bdd49939e994240a6d15dbd11559ff4 P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
I20240502 01:54:35.109086 17355 catalog_manager.cc:5484] T 3bdd49939e994240a6d15dbd11559ff4 P 94931f4b51c9493cb3d4264ce99f5e76 reported cstate change: config changed from index 4 to 5, 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "da3584f429104d23a1a8899b35cc2138" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
I20240502 01:54:35.111905 20293 raft_consensus.cc:2932] T c91f74f46b04425aa5393515cb9f781c P 84d3a4b174c34e02aedab1e620731359 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER da3584f429104d23a1a8899b35cc2138 (127.9.176.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } }
I20240502 01:54:35.114873 17497 raft_consensus.cc:2932] T c91f74f46b04425aa5393515cb9f781c P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER da3584f429104d23a1a8899b35cc2138 (127.9.176.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } }
I20240502 01:54:35.116456 17765 raft_consensus.cc:1261] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Refusing update from remote peer da3584f429104d23a1a8899b35cc2138: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:35.117367 21055 consensus_queue.cc:1035] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P da3584f429104d23a1a8899b35cc2138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:35.118413 17499 raft_consensus.cc:1261] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Refusing update from remote peer da3584f429104d23a1a8899b35cc2138: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:35.119868 20937 consensus_queue.cc:1035] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P da3584f429104d23a1a8899b35cc2138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 01:54:35.125370 17833 consensus_peers.cc:485] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P da3584f429104d23a1a8899b35cc2138 -> Peer 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200:37831): Couldn't send request to peer 94931f4b51c9493cb3d4264ce99f5e76. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 60ff2e50b1d74e0c8e0890b87f9ac8ba. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:35.126070 17343 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c91f74f46b04425aa5393515cb9f781c with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:35.129613 17351 catalog_manager.cc:5484] T c91f74f46b04425aa5393515cb9f781c P 311f765f59754bdcb4eadc04e14bd4ea reported cstate change: config changed from index 5 to 6, VOTER da3584f429104d23a1a8899b35cc2138 (127.9.176.196) evicted. New cstate: current_term: 1 leader_uuid: "84d3a4b174c34e02aedab1e620731359" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } }
I20240502 01:54:35.139333 17880 tablet_service.cc:1514] Processing DeleteTablet for tablet c91f74f46b04425aa5393515cb9f781c with delete_type TABLET_DATA_TOMBSTONED (TS da3584f429104d23a1a8899b35cc2138 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:34396
W20240502 01:54:35.142199 17700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.176.196:46381, user_credentials={real_user=slave}} blocked reactor thread for 57635.9us
I20240502 01:54:35.143860 19465 raft_consensus.cc:1261] T c91f74f46b04425aa5393515cb9f781c P 29806d3d37354c169484f6db368f8798 [term 1 FOLLOWER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:35.145287 19465 raft_consensus.cc:2932] T c91f74f46b04425aa5393515cb9f781c P 29806d3d37354c169484f6db368f8798 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 29806d3d37354c169484f6db368f8798 (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } }
I20240502 01:54:35.146188 19871 raft_consensus.cc:1261] T 37eda67598fe491bbb29aa62b2c4bec7 P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 LEARNER]: Refusing update from remote peer 311f765f59754bdcb4eadc04e14bd4ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:35.146932 20585 consensus_queue.cc:1035] T 37eda67598fe491bbb29aa62b2c4bec7 P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:35.156786 20877 tablet_copy_client.cc:308] T 974f67157aa04d3e8b49b264cfa104e8 P 29806d3d37354c169484f6db368f8798: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.194:45617
I20240502 01:54:35.157693 17654 tablet_copy_service.cc:140] P 1b50a031f2784ba498b3e52e16916205: Received BeginTabletCopySession request for tablet 974f67157aa04d3e8b49b264cfa104e8 from peer 29806d3d37354c169484f6db368f8798 ({username='slave'} at 127.9.176.197:33725)
I20240502 01:54:35.157943 17654 tablet_copy_service.cc:161] P 1b50a031f2784ba498b3e52e16916205: Beginning new tablet copy session on tablet 974f67157aa04d3e8b49b264cfa104e8 from peer 29806d3d37354c169484f6db368f8798 at {username='slave'} at 127.9.176.197:33725: session id = 29806d3d37354c169484f6db368f8798-974f67157aa04d3e8b49b264cfa104e8
I20240502 01:54:35.158691 20790 ts_tablet_manager.cc:902] T b40f968655774ce887dff8b7f4651501 P 4b7c382c9edd41babdd1a1afc904a813: Initiating tablet copy from peer 1b50a031f2784ba498b3e52e16916205 (127.9.176.194:45617)
I20240502 01:54:35.161836 17654 tablet_copy_source_session.cc:205] T 974f67157aa04d3e8b49b264cfa104e8 P 1b50a031f2784ba498b3e52e16916205: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:35.165689 19604 raft_consensus.cc:1261] T 37eda67598fe491bbb29aa62b2c4bec7 P 4b7c382c9edd41babdd1a1afc904a813 [term 1 LEARNER]: Refusing update from remote peer 311f765f59754bdcb4eadc04e14bd4ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:35.166323 17633 raft_consensus.cc:1261] T 37eda67598fe491bbb29aa62b2c4bec7 P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Refusing update from remote peer 311f765f59754bdcb4eadc04e14bd4ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:35.169754 20585 consensus_queue.cc:1035] T 37eda67598fe491bbb29aa62b2c4bec7 P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:35.170538 20788 consensus_queue.cc:1035] T 37eda67598fe491bbb29aa62b2c4bec7 P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:35.185328 19466 raft_consensus.cc:1261] T 91e0e82526c0489a8908e250bf3c851f P 29806d3d37354c169484f6db368f8798 [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:35.186899 20920 consensus_queue.cc:1035] T 91e0e82526c0489a8908e250bf3c851f P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:35.189018 19466 raft_consensus.cc:2932] T 91e0e82526c0489a8908e250bf3c851f P 29806d3d37354c169484f6db368f8798 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER da3584f429104d23a1a8899b35cc2138 (127.9.176.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } }
I20240502 01:54:35.189244 17900 consensus_queue.cc:237] T 3bdd49939e994240a6d15dbd11559ff4 P da3584f429104d23a1a8899b35cc2138 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } }
I20240502 01:54:35.192178 19872 raft_consensus.cc:1261] T 3bdd49939e994240a6d15dbd11559ff4 P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Refusing update from remote peer da3584f429104d23a1a8899b35cc2138: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:35.192863 21053 consensus_queue.cc:1035] T 3bdd49939e994240a6d15dbd11559ff4 P da3584f429104d23a1a8899b35cc2138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:35.195484 20585 raft_consensus.cc:1013] T 37eda67598fe491bbb29aa62b2c4bec7 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 LEADER]: attempt to promote peer 4b7c382c9edd41babdd1a1afc904a813: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 01:54:35.196624 21053 raft_consensus.cc:2932] T 3bdd49939e994240a6d15dbd11559ff4 P da3584f429104d23a1a8899b35cc2138 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 84d3a4b174c34e02aedab1e620731359 (127.9.176.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
I20240502 01:54:35.200815 19872 raft_consensus.cc:2932] T 3bdd49939e994240a6d15dbd11559ff4 P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 84d3a4b174c34e02aedab1e620731359 (127.9.176.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
I20240502 01:54:35.201929 20877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 974f67157aa04d3e8b49b264cfa104e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:35.207840 17340 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3bdd49939e994240a6d15dbd11559ff4 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:35.207540 17348 catalog_manager.cc:5484] T 3bdd49939e994240a6d15dbd11559ff4 P 94931f4b51c9493cb3d4264ce99f5e76 reported cstate change: config changed from index 5 to 6, VOTER 84d3a4b174c34e02aedab1e620731359 (127.9.176.195) evicted. New cstate: current_term: 1 leader_uuid: "da3584f429104d23a1a8899b35cc2138" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
I20240502 01:54:35.213645 17746 tablet_service.cc:1514] Processing DeleteTablet for tablet 3bdd49939e994240a6d15dbd11559ff4 with delete_type TABLET_DATA_TOMBSTONED (TS 84d3a4b174c34e02aedab1e620731359 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47166
I20240502 01:54:35.216603 17633 raft_consensus.cc:1261] T 3bdd49939e994240a6d15dbd11559ff4 P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Refusing update from remote peer da3584f429104d23a1a8899b35cc2138: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:35.217305 20937 consensus_queue.cc:1035] T 3bdd49939e994240a6d15dbd11559ff4 P da3584f429104d23a1a8899b35cc2138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:35.219004 17633 raft_consensus.cc:2932] T 3bdd49939e994240a6d15dbd11559ff4 P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 84d3a4b174c34e02aedab1e620731359 (127.9.176.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
I20240502 01:54:35.221212 17898 raft_consensus.cc:1261] T 768e7cbf63a94d3a815ac77a4d0acf41 P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:35.221266 19604 raft_consensus.cc:1261] T 768e7cbf63a94d3a815ac77a4d0acf41 P 4b7c382c9edd41babdd1a1afc904a813 [term 1 LEARNER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:35.222386 19736 raft_consensus.cc:1261] T 768e7cbf63a94d3a815ac77a4d0acf41 P 272b2179c1b4476090a89880df2a8baa [term 1 LEARNER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:35.223551 21007 consensus_queue.cc:1035] T c91f74f46b04425aa5393515cb9f781c P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:35.225395 21007 raft_consensus.cc:1050] T 6da59c0626c64a5988e39a831c42c268 P 84d3a4b174c34e02aedab1e620731359: attempting to promote NON_VOTER 29806d3d37354c169484f6db368f8798 to VOTER
I20240502 01:54:35.226413 21007 consensus_queue.cc:237] T 6da59c0626c64a5988e39a831c42c268 P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } }
I20240502 01:54:35.237200 21069 tablet_replica.cc:330] T 3bdd49939e994240a6d15dbd11559ff4 P 84d3a4b174c34e02aedab1e620731359: stopping tablet replica
I20240502 01:54:35.239210 20790 tablet_copy_client.cc:308] T b40f968655774ce887dff8b7f4651501 P 4b7c382c9edd41babdd1a1afc904a813: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.194:45617
I20240502 01:54:35.240347 21069 raft_consensus.cc:2229] T 3bdd49939e994240a6d15dbd11559ff4 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:35.241611 21069 raft_consensus.cc:2258] T 3bdd49939e994240a6d15dbd11559ff4 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:35.245357 21069 ts_tablet_manager.cc:1875] T 3bdd49939e994240a6d15dbd11559ff4 P 84d3a4b174c34e02aedab1e620731359: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:35.257051 20783 tablet_copy_client.cc:308] T 38a60d7db47149ff90444e5276fc8935 P 94931f4b51c9493cb3d4264ce99f5e76: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.194:45617
I20240502 01:54:35.279393 20937 raft_consensus.cc:2932] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P da3584f429104d23a1a8899b35cc2138 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } } }
I20240502 01:54:35.280871 17654 tablet_copy_service.cc:140] P 1b50a031f2784ba498b3e52e16916205: Received BeginTabletCopySession request for tablet b40f968655774ce887dff8b7f4651501 from peer 4b7c382c9edd41babdd1a1afc904a813 ({username='slave'} at 127.9.176.198:49387)
I20240502 01:54:35.281086 17654 tablet_copy_service.cc:161] P 1b50a031f2784ba498b3e52e16916205: Beginning new tablet copy session on tablet b40f968655774ce887dff8b7f4651501 from peer 4b7c382c9edd41babdd1a1afc904a813 at {username='slave'} at 127.9.176.198:49387: session id = 4b7c382c9edd41babdd1a1afc904a813-b40f968655774ce887dff8b7f4651501
I20240502 01:54:35.284974 17498 raft_consensus.cc:2932] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } } }
I20240502 01:54:35.287624 17632 raft_consensus.cc:1261] T 768e7cbf63a94d3a815ac77a4d0acf41 P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:35.288487 21024 consensus_queue.cc:1035] T 768e7cbf63a94d3a815ac77a4d0acf41 P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:35.295332 17351 catalog_manager.cc:5484] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 311f765f59754bdcb4eadc04e14bd4ea reported cstate change: config changed from index -1 to 4, NON_VOTER 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) added. New cstate: current_term: 1 leader_uuid: "da3584f429104d23a1a8899b35cc2138" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } } }
I20240502 01:54:35.297266 17654 tablet_copy_source_session.cc:205] T b40f968655774ce887dff8b7f4651501 P 1b50a031f2784ba498b3e52e16916205: Tablet Copy: opened 0 blocks and 1 log segments
W20240502 01:54:35.300228 17700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.176.197:35011, user_credentials={real_user=slave}} blocked reactor thread for 75735.5us
I20240502 01:54:35.301951 19736 raft_consensus.cc:1203] T 2c78951695284b1aaab411c0a041c28a P 272b2179c1b4476090a89880df2a8baa [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 01:54:35.304327 20790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b40f968655774ce887dff8b7f4651501. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:35.305737 17766 consensus_queue.cc:237] T f07e9b433e554c85b3c25b7e708f8b2f P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } }
I20240502 01:54:35.307905 20782 raft_consensus.cc:2932] T 37eda67598fe491bbb29aa62b2c4bec7 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
I20240502 01:54:35.311712 19872 raft_consensus.cc:2932] T 37eda67598fe491bbb29aa62b2c4bec7 P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
I20240502 01:54:35.317044 21069 ts_tablet_manager.cc:1888] T 3bdd49939e994240a6d15dbd11559ff4 P 84d3a4b174c34e02aedab1e620731359: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:54:35.320446 17354 catalog_manager.cc:5484] T 37eda67598fe491bbb29aa62b2c4bec7 P 94931f4b51c9493cb3d4264ce99f5e76 reported cstate change: config changed from index 5 to 6, 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "311f765f59754bdcb4eadc04e14bd4ea" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
I20240502 01:54:35.321101 19604 raft_consensus.cc:2932] T 37eda67598fe491bbb29aa62b2c4bec7 P 4b7c382c9edd41babdd1a1afc904a813 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
I20240502 01:54:35.325507 17633 raft_consensus.cc:2932] T 37eda67598fe491bbb29aa62b2c4bec7 P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
I20240502 01:54:35.327615 17498 consensus_queue.cc:237] T 37eda67598fe491bbb29aa62b2c4bec7 P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } }
I20240502 01:54:35.330127 21069 log.cc:1196] T 3bdd49939e994240a6d15dbd11559ff4 P 84d3a4b174c34e02aedab1e620731359: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/wal/wals/3bdd49939e994240a6d15dbd11559ff4
I20240502 01:54:35.333184 17343 catalog_manager.cc:4830] TS 84d3a4b174c34e02aedab1e620731359 (127.9.176.195:37357): tablet 3bdd49939e994240a6d15dbd11559ff4 (table rebalance_test_table_3 [id=c5e1575e337a4256a90ce5b685b3ff28]) successfully deleted
I20240502 01:54:35.337201 19871 raft_consensus.cc:1261] T 37eda67598fe491bbb29aa62b2c4bec7 P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Refusing update from remote peer 311f765f59754bdcb4eadc04e14bd4ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:35.337890 20782 consensus_queue.cc:1035] T 37eda67598fe491bbb29aa62b2c4bec7 P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:35.338347 17633 raft_consensus.cc:1261] T 37eda67598fe491bbb29aa62b2c4bec7 P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Refusing update from remote peer 311f765f59754bdcb4eadc04e14bd4ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:35.343725 20782 consensus_queue.cc:1035] T 37eda67598fe491bbb29aa62b2c4bec7 P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:35.349925 20782 raft_consensus.cc:2932] T 37eda67598fe491bbb29aa62b2c4bec7 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
I20240502 01:54:35.352416 20790 tablet_copy_client.cc:783] T b40f968655774ce887dff8b7f4651501 P 4b7c382c9edd41babdd1a1afc904a813: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:35.352150 17765 raft_consensus.cc:2932] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } } }
I20240502 01:54:35.353127 20790 tablet_copy_client.cc:647] T b40f968655774ce887dff8b7f4651501 P 4b7c382c9edd41babdd1a1afc904a813: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:35.354761 17343 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 37eda67598fe491bbb29aa62b2c4bec7 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:35.358784 19604 raft_consensus.cc:1203] T 05b67d9912f047bfb6a2bed90b82ae15 P 4b7c382c9edd41babdd1a1afc904a813 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 01:54:35.360500 19872 raft_consensus.cc:2932] T 37eda67598fe491bbb29aa62b2c4bec7 P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
I20240502 01:54:35.361418 17789 tablet_copy_service.cc:342] P 84d3a4b174c34e02aedab1e620731359: Request end of tablet copy session 94931f4b51c9493cb3d4264ce99f5e76-6da59c0626c64a5988e39a831c42c268 received from {username='slave'} at 127.9.176.200:34957
I20240502 01:54:35.361583 17789 tablet_copy_service.cc:433] P 84d3a4b174c34e02aedab1e620731359: ending tablet copy session 94931f4b51c9493cb3d4264ce99f5e76-6da59c0626c64a5988e39a831c42c268 on tablet 6da59c0626c64a5988e39a831c42c268 with peer 94931f4b51c9493cb3d4264ce99f5e76
W20240502 01:54:35.363085 17701 consensus_peers.cc:485] T f07e9b433e554c85b3c25b7e708f8b2f P 84d3a4b174c34e02aedab1e620731359 -> Peer 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200:37831): Couldn't send request to peer 94931f4b51c9493cb3d4264ce99f5e76. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f07e9b433e554c85b3c25b7e708f8b2f. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:35.364532 19467 raft_consensus.cc:2932] T c91f74f46b04425aa5393515cb9f781c P 29806d3d37354c169484f6db368f8798 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER da3584f429104d23a1a8899b35cc2138 (127.9.176.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } }
I20240502 01:54:35.365265 17499 raft_consensus.cc:1261] T f07e9b433e554c85b3c25b7e708f8b2f P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:35.365991 21024 consensus_queue.cc:1035] T f07e9b433e554c85b3c25b7e708f8b2f P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:35.370031 17633 raft_consensus.cc:1261] T f07e9b433e554c85b3c25b7e708f8b2f P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:35.370788 21024 consensus_queue.cc:1035] T f07e9b433e554c85b3c25b7e708f8b2f P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:35.373037 17498 raft_consensus.cc:1261] T 6da59c0626c64a5988e39a831c42c268 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:35.373701 21007 consensus_queue.cc:1035] T 6da59c0626c64a5988e39a831c42c268 P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:35.376070 17633 raft_consensus.cc:2932] T f07e9b433e554c85b3c25b7e708f8b2f P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } } }
I20240502 01:54:35.376703 17354 catalog_manager.cc:5484] T 37eda67598fe491bbb29aa62b2c4bec7 P 94931f4b51c9493cb3d4264ce99f5e76 reported cstate change: config changed from index 6 to 7, NON_VOTER 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) evicted. New cstate: current_term: 1 leader_uuid: "311f765f59754bdcb4eadc04e14bd4ea" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
I20240502 01:54:35.379453 20790 tablet_copy_client.cc:523] T b40f968655774ce887dff8b7f4651501 P 4b7c382c9edd41babdd1a1afc904a813: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:35.381429 21006 raft_consensus.cc:2932] T f07e9b433e554c85b3c25b7e708f8b2f P 84d3a4b174c34e02aedab1e620731359 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } } }
I20240502 01:54:35.383291 19466 raft_consensus.cc:1261] T 6da59c0626c64a5988e39a831c42c268 P 29806d3d37354c169484f6db368f8798 [term 1 LEARNER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:35.386152 19871 raft_consensus.cc:1261] T 6da59c0626c64a5988e39a831c42c268 P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 LEARNER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:35.386873 20619 consensus_queue.cc:1035] T 6da59c0626c64a5988e39a831c42c268 P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:35.395033 17498 consensus_queue.cc:237] T 37eda67598fe491bbb29aa62b2c4bec7 P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } }
I20240502 01:54:35.396414 17900 raft_consensus.cc:1261] T 6da59c0626c64a5988e39a831c42c268 P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:35.401895 17355 catalog_manager.cc:5484] T f07e9b433e554c85b3c25b7e708f8b2f P 1b50a031f2784ba498b3e52e16916205 reported cstate change: config changed from index -1 to 4, NON_VOTER 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) added. New cstate: current_term: 1 leader_uuid: "84d3a4b174c34e02aedab1e620731359" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } } }
I20240502 01:54:35.404383 21007 raft_consensus.cc:1013] T 6da59c0626c64a5988e39a831c42c268 P 84d3a4b174c34e02aedab1e620731359 [term 1 LEADER]: attempt to promote peer 94931f4b51c9493cb3d4264ce99f5e76: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 01:54:35.407347 21024 consensus_queue.cc:1035] T 768e7cbf63a94d3a815ac77a4d0acf41 P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:35.407953 21024 consensus_queue.cc:1035] T 6da59c0626c64a5988e39a831c42c268 P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:35.408434 20293 consensus_queue.cc:1035] T 768e7cbf63a94d3a815ac77a4d0acf41 P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:35.408684 17766 raft_consensus.cc:1261] T 37eda67598fe491bbb29aa62b2c4bec7 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Refusing update from remote peer 311f765f59754bdcb4eadc04e14bd4ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:35.409603 20790 tablet_bootstrap.cc:492] T b40f968655774ce887dff8b7f4651501 P 4b7c382c9edd41babdd1a1afc904a813: Bootstrap starting.
I20240502 01:54:35.410320 17632 raft_consensus.cc:2932] T 37eda67598fe491bbb29aa62b2c4bec7 P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
I20240502 01:54:35.413444 17499 raft_consensus.cc:2932] T f07e9b433e554c85b3c25b7e708f8b2f P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } } }
I20240502 01:54:35.421492 20996 ts_tablet_manager.cc:1888] T 3b3fa060e8d64911bec810ec07a358d2 P da3584f429104d23a1a8899b35cc2138: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:54:35.424129 17765 raft_consensus.cc:1261] T 37eda67598fe491bbb29aa62b2c4bec7 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Refusing update from remote peer 311f765f59754bdcb4eadc04e14bd4ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 01:54:35.424650 17700 consensus_peers.cc:485] T 88a137bcfc76410583f910b377d1be7f P 84d3a4b174c34e02aedab1e620731359 -> Peer 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198:36865): Couldn't send request to peer 4b7c382c9edd41babdd1a1afc904a813. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 88a137bcfc76410583f910b377d1be7f. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:35.427500 21006 consensus_queue.cc:1035] T 768e7cbf63a94d3a815ac77a4d0acf41 P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:35.429862 17654 tablet_copy_service.cc:140] P 1b50a031f2784ba498b3e52e16916205: Received BeginTabletCopySession request for tablet 38a60d7db47149ff90444e5276fc8935 from peer 94931f4b51c9493cb3d4264ce99f5e76 ({username='slave'} at 127.9.176.200:37843)
I20240502 01:54:35.430076 17654 tablet_copy_service.cc:161] P 1b50a031f2784ba498b3e52e16916205: Beginning new tablet copy session on tablet 38a60d7db47149ff90444e5276fc8935 from peer 94931f4b51c9493cb3d4264ce99f5e76 at {username='slave'} at 127.9.176.200:37843: session id = 94931f4b51c9493cb3d4264ce99f5e76-38a60d7db47149ff90444e5276fc8935
I20240502 01:54:35.433395 17654 tablet_copy_source_session.cc:205] T 38a60d7db47149ff90444e5276fc8935 P 1b50a031f2784ba498b3e52e16916205: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:35.435271 20783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38a60d7db47149ff90444e5276fc8935. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:35.436852 19585 tablet_service.cc:1514] Processing DeleteTablet for tablet 37eda67598fe491bbb29aa62b2c4bec7 with delete_type TABLET_DATA_TOMBSTONED (TS 4b7c382c9edd41babdd1a1afc904a813 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:58072
I20240502 01:54:35.437736 20877 tablet_copy_client.cc:783] T 974f67157aa04d3e8b49b264cfa104e8 P 29806d3d37354c169484f6db368f8798: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:35.438002 20877 tablet_copy_client.cc:647] T 974f67157aa04d3e8b49b264cfa104e8 P 29806d3d37354c169484f6db368f8798: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:35.441491 20877 tablet_copy_client.cc:523] T 974f67157aa04d3e8b49b264cfa104e8 P 29806d3d37354c169484f6db368f8798: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:35.443383 19872 raft_consensus.cc:1261] T 37eda67598fe491bbb29aa62b2c4bec7 P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Refusing update from remote peer 311f765f59754bdcb4eadc04e14bd4ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:35.444058 20788 consensus_queue.cc:1035] T 37eda67598fe491bbb29aa62b2c4bec7 P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:35.447666 20788 raft_consensus.cc:2932] T 37eda67598fe491bbb29aa62b2c4bec7 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 84d3a4b174c34e02aedab1e620731359 (127.9.176.195) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
I20240502 01:54:35.453804 17343 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 37eda67598fe491bbb29aa62b2c4bec7 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:35.454849 19871 raft_consensus.cc:2932] T 37eda67598fe491bbb29aa62b2c4bec7 P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 84d3a4b174c34e02aedab1e620731359 (127.9.176.195) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
I20240502 01:54:35.455632 20877 tablet_bootstrap.cc:492] T 974f67157aa04d3e8b49b264cfa104e8 P 29806d3d37354c169484f6db368f8798: Bootstrap starting.
I20240502 01:54:35.457441 17633 consensus_queue.cc:237] T 38a60d7db47149ff90444e5276fc8935 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } }
I20240502 01:54:35.461244 17354 catalog_manager.cc:5484] T 37eda67598fe491bbb29aa62b2c4bec7 P 94931f4b51c9493cb3d4264ce99f5e76 reported cstate change: config changed from index 7 to 8, VOTER 84d3a4b174c34e02aedab1e620731359 (127.9.176.195) evicted. New cstate: current_term: 1 leader_uuid: "311f765f59754bdcb4eadc04e14bd4ea" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
I20240502 01:54:35.465788 20920 raft_consensus.cc:1050] T df16e178e04040699201f325ce9e2979 P 1b50a031f2784ba498b3e52e16916205: attempting to promote NON_VOTER 272b2179c1b4476090a89880df2a8baa to VOTER
I20240502 01:54:35.467859 20920 consensus_queue.cc:237] T df16e178e04040699201f325ce9e2979 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } }
I20240502 01:54:35.470142 17498 raft_consensus.cc:1261] T 38a60d7db47149ff90444e5276fc8935 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240502 01:54:35.475879 17566 consensus_peers.cc:485] T 38a60d7db47149ff90444e5276fc8935 P 1b50a031f2784ba498b3e52e16916205 -> Peer 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200:37831): Couldn't send request to peer 94931f4b51c9493cb3d4264ce99f5e76. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:35.476172 17565 consensus_peers.cc:485] T 38a60d7db47149ff90444e5276fc8935 P 1b50a031f2784ba498b3e52e16916205 -> Peer 272b2179c1b4476090a89880df2a8baa (127.9.176.199:42269): Couldn't send request to peer 272b2179c1b4476090a89880df2a8baa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 38a60d7db47149ff90444e5276fc8935. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:35.476296 20911 consensus_queue.cc:1035] T 38a60d7db47149ff90444e5276fc8935 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.004s
I20240502 01:54:35.485857 21006 consensus_queue.cc:1035] T 6da59c0626c64a5988e39a831c42c268 P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:35.488461 17746 tablet_service.cc:1514] Processing DeleteTablet for tablet 37eda67598fe491bbb29aa62b2c4bec7 with delete_type TABLET_DATA_TOMBSTONED (TS 84d3a4b174c34e02aedab1e620731359 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:47166
I20240502 01:54:35.488911 21069 tablet_replica.cc:330] T 37eda67598fe491bbb29aa62b2c4bec7 P 84d3a4b174c34e02aedab1e620731359: stopping tablet replica
I20240502 01:54:35.489254 21069 raft_consensus.cc:2229] T 37eda67598fe491bbb29aa62b2c4bec7 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:35.491050 21104 tablet_replica.cc:330] T 37eda67598fe491bbb29aa62b2c4bec7 P 4b7c382c9edd41babdd1a1afc904a813: stopping tablet replica
I20240502 01:54:35.491164 21024 raft_consensus.cc:1050] T 2045e08a200148c69326351f38fd484d P 84d3a4b174c34e02aedab1e620731359: attempting to promote NON_VOTER 29806d3d37354c169484f6db368f8798 to VOTER
I20240502 01:54:35.492324 21069 raft_consensus.cc:2258] T 37eda67598fe491bbb29aa62b2c4bec7 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:35.492435 21104 raft_consensus.cc:2229] T 37eda67598fe491bbb29aa62b2c4bec7 P 4b7c382c9edd41babdd1a1afc904a813 [term 1 LEARNER]: Raft consensus shutting down.
I20240502 01:54:35.493080 21104 raft_consensus.cc:2258] T 37eda67598fe491bbb29aa62b2c4bec7 P 4b7c382c9edd41babdd1a1afc904a813 [term 1 LEARNER]: Raft consensus is shut down!
I20240502 01:54:35.494836 21069 ts_tablet_manager.cc:1875] T 37eda67598fe491bbb29aa62b2c4bec7 P 84d3a4b174c34e02aedab1e620731359: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:35.497725 17631 raft_consensus.cc:1261] T 37eda67598fe491bbb29aa62b2c4bec7 P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Refusing update from remote peer 311f765f59754bdcb4eadc04e14bd4ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:35.495328 21104 ts_tablet_manager.cc:1875] T 37eda67598fe491bbb29aa62b2c4bec7 P 4b7c382c9edd41babdd1a1afc904a813: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:35.504068 21024 consensus_queue.cc:237] T 2045e08a200148c69326351f38fd484d P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } }
I20240502 01:54:35.509768 17632 raft_consensus.cc:1261] T 2045e08a200148c69326351f38fd484d P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:35.510450 17765 raft_consensus.cc:1261] T 38a60d7db47149ff90444e5276fc8935 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:35.510648 21024 consensus_queue.cc:1035] T 2045e08a200148c69326351f38fd484d P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:35.512163 20911 consensus_queue.cc:1035] T 38a60d7db47149ff90444e5276fc8935 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:35.513023 20782 consensus_queue.cc:1035] T 37eda67598fe491bbb29aa62b2c4bec7 P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:35.513518 20783 tablet_copy_client.cc:783] T 38a60d7db47149ff90444e5276fc8935 P 94931f4b51c9493cb3d4264ce99f5e76: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:35.513718 19466 raft_consensus.cc:1261] T 2045e08a200148c69326351f38fd484d P 29806d3d37354c169484f6db368f8798 [term 1 LEARNER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:35.514214 20783 tablet_copy_client.cc:647] T 38a60d7db47149ff90444e5276fc8935 P 94931f4b51c9493cb3d4264ce99f5e76: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:35.517503 17631 raft_consensus.cc:2932] T 37eda67598fe491bbb29aa62b2c4bec7 P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 84d3a4b174c34e02aedab1e620731359 (127.9.176.195) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
I20240502 01:54:35.521162 20783 tablet_copy_client.cc:523] T 38a60d7db47149ff90444e5276fc8935 P 94931f4b51c9493cb3d4264ce99f5e76: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:35.463692 17900 consensus_queue.cc:237] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P da3584f429104d23a1a8899b35cc2138 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } }
I20240502 01:54:35.512877 17901 raft_consensus.cc:1261] T 2045e08a200148c69326351f38fd484d P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:35.523451 20996 log.cc:1196] T 3b3fa060e8d64911bec810ec07a358d2 P da3584f429104d23a1a8899b35cc2138: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/wal/wals/3b3fa060e8d64911bec810ec07a358d2
I20240502 01:54:35.524618 20996 tablet_replica.cc:330] T 91e0e82526c0489a8908e250bf3c851f P da3584f429104d23a1a8899b35cc2138: stopping tablet replica
I20240502 01:54:35.527352 20911 raft_consensus.cc:1050] T b40f968655774ce887dff8b7f4651501 P 1b50a031f2784ba498b3e52e16916205: attempting to promote NON_VOTER 272b2179c1b4476090a89880df2a8baa to VOTER
I20240502 01:54:35.527814 17340 catalog_manager.cc:4830] TS da3584f429104d23a1a8899b35cc2138 (127.9.176.196:46381): tablet 3b3fa060e8d64911bec810ec07a358d2 (table rebalance_test_table_2 [id=bf67e64a43cd4367a7420c6bf1793faa]) successfully deleted
I20240502 01:54:35.528482 21104 ts_tablet_manager.cc:1888] T 37eda67598fe491bbb29aa62b2c4bec7 P 4b7c382c9edd41babdd1a1afc904a813: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 01:54:35.528906 20911 consensus_queue.cc:237] T b40f968655774ce887dff8b7f4651501 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } }
I20240502 01:54:35.532020 19736 raft_consensus.cc:1261] T b40f968655774ce887dff8b7f4651501 P 272b2179c1b4476090a89880df2a8baa [term 1 LEARNER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:35.532927 20628 consensus_queue.cc:1035] T b40f968655774ce887dff8b7f4651501 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240502 01:54:35.535759 17565 consensus_peers.cc:485] T b40f968655774ce887dff8b7f4651501 P 1b50a031f2784ba498b3e52e16916205 -> Peer 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198:36865): Couldn't send request to peer 4b7c382c9edd41babdd1a1afc904a813. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:35.535439 20635 raft_consensus.cc:2932] T 38a60d7db47149ff90444e5276fc8935 P 1b50a031f2784ba498b3e52e16916205 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 272b2179c1b4476090a89880df2a8baa (127.9.176.199) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } } }
I20240502 01:54:35.537851 17499 raft_consensus.cc:1261] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Refusing update from remote peer da3584f429104d23a1a8899b35cc2138: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:35.542869 19736 raft_consensus.cc:1261] T df16e178e04040699201f325ce9e2979 P 272b2179c1b4476090a89880df2a8baa [term 1 LEARNER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:35.543624 20628 consensus_queue.cc:1035] T df16e178e04040699201f325ce9e2979 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:35.548281 17350 catalog_manager.cc:5484] T 38a60d7db47149ff90444e5276fc8935 P 1b50a031f2784ba498b3e52e16916205 reported cstate change: config changed from index 4 to 5, NON_VOTER 272b2179c1b4476090a89880df2a8baa (127.9.176.199) added. New cstate: current_term: 1 leader_uuid: "1b50a031f2784ba498b3e52e16916205" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:35.553740 17497 raft_consensus.cc:1261] T df16e178e04040699201f325ce9e2979 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:35.554455 20635 consensus_queue.cc:1035] T df16e178e04040699201f325ce9e2979 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:35.556071 17499 raft_consensus.cc:2932] T 38a60d7db47149ff90444e5276fc8935 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 272b2179c1b4476090a89880df2a8baa (127.9.176.199) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } } }
I20240502 01:54:35.561230 20783 tablet_bootstrap.cc:492] T 38a60d7db47149ff90444e5276fc8935 P 94931f4b51c9493cb3d4264ce99f5e76: Bootstrap starting.
W20240502 01:54:35.527261 17833 consensus_peers.cc:485] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P da3584f429104d23a1a8899b35cc2138 -> Peer 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200:37831): Couldn't send request to peer 94931f4b51c9493cb3d4264ce99f5e76. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 60ff2e50b1d74e0c8e0890b87f9ac8ba. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:35.532773 17900 raft_consensus.cc:1261] T b40f968655774ce887dff8b7f4651501 P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:35.562114 20996 raft_consensus.cc:2229] T 91e0e82526c0489a8908e250bf3c851f P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:35.544489 21053 consensus_queue.cc:1035] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P da3584f429104d23a1a8899b35cc2138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:35.563367 20911 consensus_queue.cc:1035] T b40f968655774ce887dff8b7f4651501 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:35.564749 20996 raft_consensus.cc:2258] T 91e0e82526c0489a8908e250bf3c851f P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:35.567694 20628 raft_consensus.cc:2932] T b40f968655774ce887dff8b7f4651501 P 1b50a031f2784ba498b3e52e16916205 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 272b2179c1b4476090a89880df2a8baa (127.9.176.199) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } } }
I20240502 01:54:35.576498 19736 raft_consensus.cc:2932] T b40f968655774ce887dff8b7f4651501 P 272b2179c1b4476090a89880df2a8baa [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 272b2179c1b4476090a89880df2a8baa (127.9.176.199) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } } }
I20240502 01:54:35.577585 17900 raft_consensus.cc:2932] T b40f968655774ce887dff8b7f4651501 P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 272b2179c1b4476090a89880df2a8baa (127.9.176.199) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } } }
I20240502 01:54:35.586258 17350 catalog_manager.cc:5484] T b40f968655774ce887dff8b7f4651501 P 1b50a031f2784ba498b3e52e16916205 reported cstate change: config changed from index 5 to 6, 272b2179c1b4476090a89880df2a8baa (127.9.176.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1b50a031f2784ba498b3e52e16916205" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:35.593494 20996 ts_tablet_manager.cc:1875] T 91e0e82526c0489a8908e250bf3c851f P da3584f429104d23a1a8899b35cc2138: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:35.603036 21007 consensus_queue.cc:1035] T 2045e08a200148c69326351f38fd484d P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.077s
I20240502 01:54:35.604178 20293 raft_consensus.cc:2932] T 6da59c0626c64a5988e39a831c42c268 P 84d3a4b174c34e02aedab1e620731359 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 29806d3d37354c169484f6db368f8798 (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } } }
I20240502 01:54:35.604838 17764 raft_consensus.cc:1261] T b40f968655774ce887dff8b7f4651501 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:35.605118 21053 raft_consensus.cc:2932] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P da3584f429104d23a1a8899b35cc2138 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 272b2179c1b4476090a89880df2a8baa (127.9.176.199) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } } }
I20240502 01:54:35.608827 21024 consensus_queue.cc:1035] T 2045e08a200148c69326351f38fd484d P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.005s
I20240502 01:54:35.610083 17766 raft_consensus.cc:1261] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Refusing update from remote peer da3584f429104d23a1a8899b35cc2138: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:35.614359 17350 catalog_manager.cc:5484] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P da3584f429104d23a1a8899b35cc2138 reported cstate change: config changed from index 4 to 5, NON_VOTER 272b2179c1b4476090a89880df2a8baa (127.9.176.199) added. New cstate: current_term: 1 leader_uuid: "da3584f429104d23a1a8899b35cc2138" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:35.615077 21053 consensus_queue.cc:1035] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P da3584f429104d23a1a8899b35cc2138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 01:54:35.616250 21111 log.cc:924] Time spent T df16e178e04040699201f325ce9e2979 P 1b50a031f2784ba498b3e52e16916205: Append to log took a long time: real 0.077s	user 0.000s	sys 0.000s
I20240502 01:54:35.618345 17766 raft_consensus.cc:2932] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 272b2179c1b4476090a89880df2a8baa (127.9.176.199) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } } }
I20240502 01:54:35.630122 17499 raft_consensus.cc:2932] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 272b2179c1b4476090a89880df2a8baa (127.9.176.199) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } } }
I20240502 01:54:35.634579 20877 tablet_bootstrap.cc:492] T 974f67157aa04d3e8b49b264cfa104e8 P 29806d3d37354c169484f6db368f8798: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:35.635053 20877 tablet_bootstrap.cc:492] T 974f67157aa04d3e8b49b264cfa104e8 P 29806d3d37354c169484f6db368f8798: Bootstrap complete.
I20240502 01:54:35.634572 19871 raft_consensus.cc:2932] T 6da59c0626c64a5988e39a831c42c268 P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 29806d3d37354c169484f6db368f8798 (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } } }
I20240502 01:54:35.635349 20877 ts_tablet_manager.cc:1367] T 974f67157aa04d3e8b49b264cfa104e8 P 29806d3d37354c169484f6db368f8798: Time spent bootstrapping tablet: real 0.180s	user 0.047s	sys 0.005s
I20240502 01:54:35.636360 20877 raft_consensus.cc:348] T 974f67157aa04d3e8b49b264cfa104e8 P 29806d3d37354c169484f6db368f8798 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } }
I20240502 01:54:35.636618 20877 raft_consensus.cc:729] T 974f67157aa04d3e8b49b264cfa104e8 P 29806d3d37354c169484f6db368f8798 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 29806d3d37354c169484f6db368f8798, State: Initialized, Role: LEARNER
I20240502 01:54:35.636863 20877 consensus_queue.cc:260] T 974f67157aa04d3e8b49b264cfa104e8 P 29806d3d37354c169484f6db368f8798 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } }
I20240502 01:54:35.641609 20996 ts_tablet_manager.cc:1888] T 91e0e82526c0489a8908e250bf3c851f P da3584f429104d23a1a8899b35cc2138: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:54:35.641428 17350 catalog_manager.cc:5484] T 6da59c0626c64a5988e39a831c42c268 P 94931f4b51c9493cb3d4264ce99f5e76 reported cstate change: config changed from index 5 to 6, 29806d3d37354c169484f6db368f8798 (127.9.176.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "84d3a4b174c34e02aedab1e620731359" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } } }
I20240502 01:54:35.643319 21007 raft_consensus.cc:1050] T 6da59c0626c64a5988e39a831c42c268 P 84d3a4b174c34e02aedab1e620731359: attempting to promote NON_VOTER 94931f4b51c9493cb3d4264ce99f5e76 to VOTER
I20240502 01:54:35.637478 20877 ts_tablet_manager.cc:1398] T 974f67157aa04d3e8b49b264cfa104e8 P 29806d3d37354c169484f6db368f8798: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 01:54:35.644258 21007 consensus_queue.cc:237] T 6da59c0626c64a5988e39a831c42c268 P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } }
I20240502 01:54:35.644798 17654 tablet_copy_service.cc:342] P 1b50a031f2784ba498b3e52e16916205: Request end of tablet copy session 29806d3d37354c169484f6db368f8798-974f67157aa04d3e8b49b264cfa104e8 received from {username='slave'} at 127.9.176.197:33725
I20240502 01:54:35.644963 17654 tablet_copy_service.cc:433] P 1b50a031f2784ba498b3e52e16916205: ending tablet copy session 29806d3d37354c169484f6db368f8798-974f67157aa04d3e8b49b264cfa104e8 on tablet 974f67157aa04d3e8b49b264cfa104e8 with peer 29806d3d37354c169484f6db368f8798
I20240502 01:54:35.647486 20996 log.cc:1196] T 91e0e82526c0489a8908e250bf3c851f P da3584f429104d23a1a8899b35cc2138: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/wal/wals/91e0e82526c0489a8908e250bf3c851f
I20240502 01:54:35.648401 17340 catalog_manager.cc:4830] TS da3584f429104d23a1a8899b35cc2138 (127.9.176.196:46381): tablet 91e0e82526c0489a8908e250bf3c851f (table rebalance_test_table_2 [id=bf67e64a43cd4367a7420c6bf1793faa]) successfully deleted
I20240502 01:54:35.632949 20790 tablet_bootstrap.cc:492] T b40f968655774ce887dff8b7f4651501 P 4b7c382c9edd41babdd1a1afc904a813: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:35.649912 21104 log.cc:1196] T 37eda67598fe491bbb29aa62b2c4bec7 P 4b7c382c9edd41babdd1a1afc904a813: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/wal/wals/37eda67598fe491bbb29aa62b2c4bec7
I20240502 01:54:35.650418 17499 raft_consensus.cc:1261] T 6da59c0626c64a5988e39a831c42c268 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:35.648381 20996 tablet_replica.cc:330] T c91f74f46b04425aa5393515cb9f781c P da3584f429104d23a1a8899b35cc2138: stopping tablet replica
I20240502 01:54:35.650343 20790 tablet_bootstrap.cc:492] T b40f968655774ce887dff8b7f4651501 P 4b7c382c9edd41babdd1a1afc904a813: Bootstrap complete.
I20240502 01:54:35.651010 20996 raft_consensus.cc:2229] T c91f74f46b04425aa5393515cb9f781c P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:35.651166 20790 ts_tablet_manager.cc:1367] T b40f968655774ce887dff8b7f4651501 P 4b7c382c9edd41babdd1a1afc904a813: Time spent bootstrapping tablet: real 0.242s	user 0.050s	sys 0.008s
I20240502 01:54:35.651842 17340 catalog_manager.cc:4830] TS 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198:36865): tablet 37eda67598fe491bbb29aa62b2c4bec7 (table rebalance_test_table_1 [id=952912d7ec8d493bbb8985fffe9aa918]) successfully deleted
I20240502 01:54:35.652206 20790 raft_consensus.cc:348] T b40f968655774ce887dff8b7f4651501 P 4b7c382c9edd41babdd1a1afc904a813 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } }
I20240502 01:54:35.652509 20790 raft_consensus.cc:729] T b40f968655774ce887dff8b7f4651501 P 4b7c382c9edd41babdd1a1afc904a813 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4b7c382c9edd41babdd1a1afc904a813, State: Initialized, Role: LEARNER
I20240502 01:54:35.652777 20790 consensus_queue.cc:260] T b40f968655774ce887dff8b7f4651501 P 4b7c382c9edd41babdd1a1afc904a813 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } }
I20240502 01:54:35.653316 21006 consensus_queue.cc:1035] T 6da59c0626c64a5988e39a831c42c268 P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:35.653568 20790 ts_tablet_manager.cc:1398] T b40f968655774ce887dff8b7f4651501 P 4b7c382c9edd41babdd1a1afc904a813: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 01:54:35.654422 20996 raft_consensus.cc:2258] T c91f74f46b04425aa5393515cb9f781c P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:35.654439 17654 tablet_copy_service.cc:342] P 1b50a031f2784ba498b3e52e16916205: Request end of tablet copy session 4b7c382c9edd41babdd1a1afc904a813-b40f968655774ce887dff8b7f4651501 received from {username='slave'} at 127.9.176.198:49387
I20240502 01:54:35.654582 17654 tablet_copy_service.cc:433] P 1b50a031f2784ba498b3e52e16916205: ending tablet copy session 4b7c382c9edd41babdd1a1afc904a813-b40f968655774ce887dff8b7f4651501 on tablet b40f968655774ce887dff8b7f4651501 with peer 4b7c382c9edd41babdd1a1afc904a813
I20240502 01:54:35.655217 17499 raft_consensus.cc:2932] T 6da59c0626c64a5988e39a831c42c268 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 29806d3d37354c169484f6db368f8798 (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } } }
I20240502 01:54:35.656387 19871 raft_consensus.cc:1261] T 6da59c0626c64a5988e39a831c42c268 P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 LEARNER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:35.657217 20293 consensus_queue.cc:1035] T 6da59c0626c64a5988e39a831c42c268 P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:35.660010 20996 ts_tablet_manager.cc:1875] T c91f74f46b04425aa5393515cb9f781c P da3584f429104d23a1a8899b35cc2138: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:35.660907 20790 ts_tablet_manager.cc:902] T 88a137bcfc76410583f910b377d1be7f P 4b7c382c9edd41babdd1a1afc904a813: Initiating tablet copy from peer 84d3a4b174c34e02aedab1e620731359 (127.9.176.195:37357)
I20240502 01:54:35.660887 20911 consensus_queue.cc:1035] T b40f968655774ce887dff8b7f4651501 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:35.663527 19466 raft_consensus.cc:1261] T 6da59c0626c64a5988e39a831c42c268 P 29806d3d37354c169484f6db368f8798 [term 1 FOLLOWER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:35.665040 21006 raft_consensus.cc:2932] T 768e7cbf63a94d3a815ac77a4d0acf41 P 84d3a4b174c34e02aedab1e620731359 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 272b2179c1b4476090a89880df2a8baa (127.9.176.199) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } } }
I20240502 01:54:35.670528 17632 raft_consensus.cc:2932] T 768e7cbf63a94d3a815ac77a4d0acf41 P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 272b2179c1b4476090a89880df2a8baa (127.9.176.199) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } } }
W20240502 01:54:35.677282 17832 consensus_peers.cc:485] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P da3584f429104d23a1a8899b35cc2138 -> Peer 272b2179c1b4476090a89880df2a8baa (127.9.176.199:42269): Couldn't send request to peer 272b2179c1b4476090a89880df2a8baa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 60ff2e50b1d74e0c8e0890b87f9ac8ba. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:35.677858 17355 catalog_manager.cc:5484] T 768e7cbf63a94d3a815ac77a4d0acf41 P 1b50a031f2784ba498b3e52e16916205 reported cstate change: config changed from index 5 to 6, 272b2179c1b4476090a89880df2a8baa (127.9.176.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "84d3a4b174c34e02aedab1e620731359" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } } }
I20240502 01:54:35.680302 20619 raft_consensus.cc:2932] T 6da59c0626c64a5988e39a831c42c268 P 84d3a4b174c34e02aedab1e620731359 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
I20240502 01:54:35.692677 17901 raft_consensus.cc:1261] T 6da59c0626c64a5988e39a831c42c268 P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:35.694120 17766 raft_consensus.cc:1261] T df16e178e04040699201f325ce9e2979 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:35.693753 19871 raft_consensus.cc:2932] T 6da59c0626c64a5988e39a831c42c268 P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
I20240502 01:54:35.699446 20628 consensus_queue.cc:1035] T df16e178e04040699201f325ce9e2979 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:35.700352 17355 catalog_manager.cc:5484] T 6da59c0626c64a5988e39a831c42c268 P 94931f4b51c9493cb3d4264ce99f5e76 reported cstate change: config changed from index 6 to 7, 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "84d3a4b174c34e02aedab1e620731359" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
I20240502 01:54:35.708753 20996 ts_tablet_manager.cc:1888] T c91f74f46b04425aa5393515cb9f781c P da3584f429104d23a1a8899b35cc2138: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:54:35.709301 20293 consensus_queue.cc:1035] T 6da59c0626c64a5988e39a831c42c268 P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:35.711584 21006 raft_consensus.cc:2932] T 2045e08a200148c69326351f38fd484d P 84d3a4b174c34e02aedab1e620731359 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 29806d3d37354c169484f6db368f8798 (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } }
I20240502 01:54:35.714211 20996 log.cc:1196] T c91f74f46b04425aa5393515cb9f781c P da3584f429104d23a1a8899b35cc2138: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/wal/wals/c91f74f46b04425aa5393515cb9f781c
I20240502 01:54:35.696990 17900 raft_consensus.cc:2932] T 6da59c0626c64a5988e39a831c42c268 P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 29806d3d37354c169484f6db368f8798 (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } } }
I20240502 01:54:35.722738 17340 catalog_manager.cc:4830] TS da3584f429104d23a1a8899b35cc2138 (127.9.176.196:46381): tablet c91f74f46b04425aa5393515cb9f781c (table rebalance_test_table_2 [id=bf67e64a43cd4367a7420c6bf1793faa]) successfully deleted
I20240502 01:54:35.726570 21069 ts_tablet_manager.cc:1888] T 37eda67598fe491bbb29aa62b2c4bec7 P 84d3a4b174c34e02aedab1e620731359: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:54:35.726897 21069 log.cc:1196] T 37eda67598fe491bbb29aa62b2c4bec7 P 84d3a4b174c34e02aedab1e620731359: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/wal/wals/37eda67598fe491bbb29aa62b2c4bec7
I20240502 01:54:35.729099 17499 raft_consensus.cc:2932] T 6da59c0626c64a5988e39a831c42c268 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
I20240502 01:54:35.743664 21007 raft_consensus.cc:1050] T 768e7cbf63a94d3a815ac77a4d0acf41 P 84d3a4b174c34e02aedab1e620731359: attempting to promote NON_VOTER 4b7c382c9edd41babdd1a1afc904a813 to VOTER
I20240502 01:54:35.744565 21007 consensus_queue.cc:237] T 768e7cbf63a94d3a815ac77a4d0acf41 P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } }
I20240502 01:54:35.745327 20790 tablet_copy_client.cc:308] T 88a137bcfc76410583f910b377d1be7f P 4b7c382c9edd41babdd1a1afc904a813: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.195:37357
I20240502 01:54:35.748356 17343 catalog_manager.cc:4830] TS 84d3a4b174c34e02aedab1e620731359 (127.9.176.195:37357): tablet 37eda67598fe491bbb29aa62b2c4bec7 (table rebalance_test_table_1 [id=952912d7ec8d493bbb8985fffe9aa918]) successfully deleted
I20240502 01:54:35.750730 20628 raft_consensus.cc:1050] T 2c78951695284b1aaab411c0a041c28a P 1b50a031f2784ba498b3e52e16916205: attempting to promote NON_VOTER 272b2179c1b4476090a89880df2a8baa to VOTER
I20240502 01:54:35.751550 20628 consensus_queue.cc:237] T 2c78951695284b1aaab411c0a041c28a P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } }
I20240502 01:54:35.766570 20937 raft_consensus.cc:1050] T 05b67d9912f047bfb6a2bed90b82ae15 P da3584f429104d23a1a8899b35cc2138: attempting to promote NON_VOTER 4b7c382c9edd41babdd1a1afc904a813 to VOTER
I20240502 01:54:35.767897 20937 consensus_queue.cc:237] T 05b67d9912f047bfb6a2bed90b82ae15 P da3584f429104d23a1a8899b35cc2138 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } }
I20240502 01:54:35.771382 20783 tablet_bootstrap.cc:492] T 38a60d7db47149ff90444e5276fc8935 P 94931f4b51c9493cb3d4264ce99f5e76: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:35.771445 17632 raft_consensus.cc:1261] T 05b67d9912f047bfb6a2bed90b82ae15 P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Refusing update from remote peer da3584f429104d23a1a8899b35cc2138: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:35.771960 20783 tablet_bootstrap.cc:492] T 38a60d7db47149ff90444e5276fc8935 P 94931f4b51c9493cb3d4264ce99f5e76: Bootstrap complete.
I20240502 01:54:35.772099 17765 raft_consensus.cc:1261] T 05b67d9912f047bfb6a2bed90b82ae15 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Refusing update from remote peer da3584f429104d23a1a8899b35cc2138: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:35.772281 20783 ts_tablet_manager.cc:1367] T 38a60d7db47149ff90444e5276fc8935 P 94931f4b51c9493cb3d4264ce99f5e76: Time spent bootstrapping tablet: real 0.211s	user 0.047s	sys 0.008s
I20240502 01:54:35.773433 21055 consensus_queue.cc:1035] T 05b67d9912f047bfb6a2bed90b82ae15 P da3584f429104d23a1a8899b35cc2138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:35.774008 20783 raft_consensus.cc:348] T 38a60d7db47149ff90444e5276fc8935 P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } }
I20240502 01:54:35.774181 21055 consensus_queue.cc:1035] T 05b67d9912f047bfb6a2bed90b82ae15 P da3584f429104d23a1a8899b35cc2138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:35.774299 20783 raft_consensus.cc:729] T 38a60d7db47149ff90444e5276fc8935 P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 94931f4b51c9493cb3d4264ce99f5e76, State: Initialized, Role: LEARNER
I20240502 01:54:35.774554 20783 consensus_queue.cc:260] T 38a60d7db47149ff90444e5276fc8935 P 94931f4b51c9493cb3d4264ce99f5e76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } }
I20240502 01:54:35.775733 19466 raft_consensus.cc:2932] T 6da59c0626c64a5988e39a831c42c268 P 29806d3d37354c169484f6db368f8798 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 29806d3d37354c169484f6db368f8798 (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } } }
I20240502 01:54:35.776597 20783 ts_tablet_manager.cc:1398] T 38a60d7db47149ff90444e5276fc8935 P 94931f4b51c9493cb3d4264ce99f5e76: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.001s
I20240502 01:54:35.776921 17789 tablet_copy_service.cc:140] P 84d3a4b174c34e02aedab1e620731359: Received BeginTabletCopySession request for tablet 88a137bcfc76410583f910b377d1be7f from peer 4b7c382c9edd41babdd1a1afc904a813 ({username='slave'} at 127.9.176.198:55177)
I20240502 01:54:35.777088 17789 tablet_copy_service.cc:161] P 84d3a4b174c34e02aedab1e620731359: Beginning new tablet copy session on tablet 88a137bcfc76410583f910b377d1be7f from peer 4b7c382c9edd41babdd1a1afc904a813 at {username='slave'} at 127.9.176.198:55177: session id = 4b7c382c9edd41babdd1a1afc904a813-88a137bcfc76410583f910b377d1be7f
I20240502 01:54:35.777303 17654 tablet_copy_service.cc:342] P 1b50a031f2784ba498b3e52e16916205: Request end of tablet copy session 94931f4b51c9493cb3d4264ce99f5e76-38a60d7db47149ff90444e5276fc8935 received from {username='slave'} at 127.9.176.200:37843
I20240502 01:54:35.777431 17654 tablet_copy_service.cc:433] P 1b50a031f2784ba498b3e52e16916205: ending tablet copy session 94931f4b51c9493cb3d4264ce99f5e76-38a60d7db47149ff90444e5276fc8935 on tablet 38a60d7db47149ff90444e5276fc8935 with peer 94931f4b51c9493cb3d4264ce99f5e76
I20240502 01:54:35.778584 21024 consensus_queue.cc:1035] T 6da59c0626c64a5988e39a831c42c268 P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:35.780108 20879 ts_tablet_manager.cc:902] T f07e9b433e554c85b3c25b7e708f8b2f P 94931f4b51c9493cb3d4264ce99f5e76: Initiating tablet copy from peer 84d3a4b174c34e02aedab1e620731359 (127.9.176.195:37357)
I20240502 01:54:35.780639 20911 raft_consensus.cc:2932] T df16e178e04040699201f325ce9e2979 P 1b50a031f2784ba498b3e52e16916205 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 272b2179c1b4476090a89880df2a8baa (127.9.176.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } }
I20240502 01:54:35.781926 19605 raft_consensus.cc:1261] T 768e7cbf63a94d3a815ac77a4d0acf41 P 4b7c382c9edd41babdd1a1afc904a813 [term 1 LEARNER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:35.783083 21140 consensus_queue.cc:1035] T 768e7cbf63a94d3a815ac77a4d0acf41 P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:35.783157 19467 raft_consensus.cc:2932] T 2045e08a200148c69326351f38fd484d P 29806d3d37354c169484f6db368f8798 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 29806d3d37354c169484f6db368f8798 (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } }
I20240502 01:54:35.783972 19604 raft_consensus.cc:2932] T 768e7cbf63a94d3a815ac77a4d0acf41 P 4b7c382c9edd41babdd1a1afc904a813 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 272b2179c1b4476090a89880df2a8baa (127.9.176.199) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } } }
I20240502 01:54:35.786477 19738 raft_consensus.cc:1261] T 768e7cbf63a94d3a815ac77a4d0acf41 P 272b2179c1b4476090a89880df2a8baa [term 1 FOLLOWER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:35.788159 17632 raft_consensus.cc:1261] T 768e7cbf63a94d3a815ac77a4d0acf41 P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:35.789196 19466 raft_consensus.cc:2932] T 6da59c0626c64a5988e39a831c42c268 P 29806d3d37354c169484f6db368f8798 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
I20240502 01:54:35.795100 17789 tablet_copy_source_session.cc:205] T 88a137bcfc76410583f910b377d1be7f P 84d3a4b174c34e02aedab1e620731359: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:35.801926 17354 catalog_manager.cc:5484] T df16e178e04040699201f325ce9e2979 P 1b50a031f2784ba498b3e52e16916205 reported cstate change: config changed from index 4 to 5, 272b2179c1b4476090a89880df2a8baa (127.9.176.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1b50a031f2784ba498b3e52e16916205" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:35.800137 21141 consensus_queue.cc:1035] T 768e7cbf63a94d3a815ac77a4d0acf41 P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.002s
I20240502 01:54:35.802095 17355 catalog_manager.cc:5484] T 2045e08a200148c69326351f38fd484d P 29806d3d37354c169484f6db368f8798 reported cstate change: config changed from index 4 to 5, 29806d3d37354c169484f6db368f8798 (127.9.176.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "84d3a4b174c34e02aedab1e620731359" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } }
I20240502 01:54:35.805003 19736 raft_consensus.cc:2932] T 768e7cbf63a94d3a815ac77a4d0acf41 P 272b2179c1b4476090a89880df2a8baa [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 272b2179c1b4476090a89880df2a8baa (127.9.176.199) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } } }
I20240502 01:54:35.807389 21024 consensus_queue.cc:1035] T 768e7cbf63a94d3a815ac77a4d0acf41 P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:35.812760 20790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88a137bcfc76410583f910b377d1be7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:35.817224 21024 raft_consensus.cc:2932] T 768e7cbf63a94d3a815ac77a4d0acf41 P 84d3a4b174c34e02aedab1e620731359 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
I20240502 01:54:35.820399 19738 raft_consensus.cc:2932] T df16e178e04040699201f325ce9e2979 P 272b2179c1b4476090a89880df2a8baa [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 272b2179c1b4476090a89880df2a8baa (127.9.176.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } }
I20240502 01:54:35.822379 20790 tablet_copy_client.cc:783] T 88a137bcfc76410583f910b377d1be7f P 4b7c382c9edd41babdd1a1afc904a813: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:35.822747 20790 tablet_copy_client.cc:647] T 88a137bcfc76410583f910b377d1be7f P 4b7c382c9edd41babdd1a1afc904a813: tablet copy: Starting download of 1 WAL segments...
W20240502 01:54:35.829670 17832 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.176.198:36865, user_credentials={real_user=slave}} blocked reactor thread for 63672.5us
I20240502 01:54:35.830276 17500 consensus_queue.cc:237] T 37eda67598fe491bbb29aa62b2c4bec7 P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } }
I20240502 01:54:35.832178 19605 raft_consensus.cc:1261] T 05b67d9912f047bfb6a2bed90b82ae15 P 4b7c382c9edd41babdd1a1afc904a813 [term 1 LEARNER]: Refusing update from remote peer da3584f429104d23a1a8899b35cc2138: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:35.832752 17901 raft_consensus.cc:2932] T 768e7cbf63a94d3a815ac77a4d0acf41 P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 272b2179c1b4476090a89880df2a8baa (127.9.176.199) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } } }
I20240502 01:54:35.837553 17631 raft_consensus.cc:1261] T 37eda67598fe491bbb29aa62b2c4bec7 P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Refusing update from remote peer 311f765f59754bdcb4eadc04e14bd4ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:54:35.838621 19871 raft_consensus.cc:1261] T 37eda67598fe491bbb29aa62b2c4bec7 P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Refusing update from remote peer 311f765f59754bdcb4eadc04e14bd4ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:54:35.839967 20782 consensus_queue.cc:1035] T 37eda67598fe491bbb29aa62b2c4bec7 P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:35.842850 20782 consensus_queue.cc:1035] T 37eda67598fe491bbb29aa62b2c4bec7 P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:35.843871 20782 raft_consensus.cc:2932] T 37eda67598fe491bbb29aa62b2c4bec7 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 29806d3d37354c169484f6db368f8798 (127.9.176.197) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } } }
I20240502 01:54:35.847180 17631 raft_consensus.cc:2932] T 37eda67598fe491bbb29aa62b2c4bec7 P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 29806d3d37354c169484f6db368f8798 (127.9.176.197) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } } }
I20240502 01:54:35.848455 19871 raft_consensus.cc:2932] T 37eda67598fe491bbb29aa62b2c4bec7 P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 29806d3d37354c169484f6db368f8798 (127.9.176.197) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } } }
I20240502 01:54:35.851227 17897 raft_consensus.cc:2932] T 6da59c0626c64a5988e39a831c42c268 P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
I20240502 01:54:35.855628 17899 raft_consensus.cc:1261] T 768e7cbf63a94d3a815ac77a4d0acf41 P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:35.863896 17497 raft_consensus.cc:2932] T df16e178e04040699201f325ce9e2979 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 272b2179c1b4476090a89880df2a8baa (127.9.176.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } }
I20240502 01:54:35.869824 19736 raft_consensus.cc:1261] T 2c78951695284b1aaab411c0a041c28a P 272b2179c1b4476090a89880df2a8baa [term 1 LEARNER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:35.870851 20635 consensus_queue.cc:1035] T 2c78951695284b1aaab411c0a041c28a P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:35.876209 20937 consensus_queue.cc:1035] T 05b67d9912f047bfb6a2bed90b82ae15 P da3584f429104d23a1a8899b35cc2138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:35.878767 17500 raft_consensus.cc:1261] T 2c78951695284b1aaab411c0a041c28a P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:35.879832 20911 consensus_queue.cc:1035] T 2c78951695284b1aaab411c0a041c28a P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:35.903421 17631 raft_consensus.cc:2932] T 768e7cbf63a94d3a815ac77a4d0acf41 P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
I20240502 01:54:35.904235 19737 raft_consensus.cc:2932] T 768e7cbf63a94d3a815ac77a4d0acf41 P 272b2179c1b4476090a89880df2a8baa [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
I20240502 01:54:35.911631 17613 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:54:35.918216 17348 catalog_manager.cc:5484] T 37eda67598fe491bbb29aa62b2c4bec7 P 1b50a031f2784ba498b3e52e16916205 reported cstate change: config changed from index 8 to 9, NON_VOTER 29806d3d37354c169484f6db368f8798 (127.9.176.197) added. New cstate: current_term: 1 leader_uuid: "311f765f59754bdcb4eadc04e14bd4ea" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } } }
I20240502 01:54:35.931227 17348 catalog_manager.cc:5484] T 768e7cbf63a94d3a815ac77a4d0acf41 P 272b2179c1b4476090a89880df2a8baa reported cstate change: config changed from index 6 to 7, 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "84d3a4b174c34e02aedab1e620731359" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
I20240502 01:54:35.945075 17763 raft_consensus.cc:1261] T 2c78951695284b1aaab411c0a041c28a P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:35.953181 20635 consensus_queue.cc:1035] T 2c78951695284b1aaab411c0a041c28a P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:35.954023 20635 raft_consensus.cc:2932] T 2c78951695284b1aaab411c0a041c28a P 1b50a031f2784ba498b3e52e16916205 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 272b2179c1b4476090a89880df2a8baa (127.9.176.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } }
I20240502 01:54:35.962440 17498 raft_consensus.cc:2932] T 2c78951695284b1aaab411c0a041c28a P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 272b2179c1b4476090a89880df2a8baa (127.9.176.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } }
I20240502 01:54:35.965051 19718 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:35.965884 20293 consensus_queue.cc:1035] T 768e7cbf63a94d3a815ac77a4d0acf41 P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.015s
I20240502 01:54:35.973316 19604 raft_consensus.cc:2932] T 768e7cbf63a94d3a815ac77a4d0acf41 P 4b7c382c9edd41babdd1a1afc904a813 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
I20240502 01:54:35.997968 20790 tablet_copy_client.cc:523] T 88a137bcfc76410583f910b377d1be7f P 4b7c382c9edd41babdd1a1afc904a813: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:36.001864 17348 catalog_manager.cc:5484] T 2c78951695284b1aaab411c0a041c28a P 1b50a031f2784ba498b3e52e16916205 reported cstate change: config changed from index 4 to 5, 272b2179c1b4476090a89880df2a8baa (127.9.176.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1b50a031f2784ba498b3e52e16916205" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:36.009847 17762 consensus_queue.cc:237] T 768e7cbf63a94d3a815ac77a4d0acf41 P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } }
I20240502 01:54:36.013288 17477 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:54:36.016300 19604 raft_consensus.cc:1261] T 768e7cbf63a94d3a815ac77a4d0acf41 P 4b7c382c9edd41babdd1a1afc904a813 [term 1 FOLLOWER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:36.017057 21143 consensus_queue.cc:1035] T 768e7cbf63a94d3a815ac77a4d0acf41 P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:36.020586 19738 raft_consensus.cc:1261] T 768e7cbf63a94d3a815ac77a4d0acf41 P 272b2179c1b4476090a89880df2a8baa [term 1 FOLLOWER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:36.021358 20783 ts_tablet_manager.cc:902] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 94931f4b51c9493cb3d4264ce99f5e76: Initiating tablet copy from peer da3584f429104d23a1a8899b35cc2138 (127.9.176.196:46381)
I20240502 01:54:36.022467 20937 raft_consensus.cc:2932] T 05b67d9912f047bfb6a2bed90b82ae15 P da3584f429104d23a1a8899b35cc2138 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
I20240502 01:54:36.027083 17762 raft_consensus.cc:2932] T 05b67d9912f047bfb6a2bed90b82ae15 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
I20240502 01:54:36.033363 17350 catalog_manager.cc:5484] T 05b67d9912f047bfb6a2bed90b82ae15 P da3584f429104d23a1a8899b35cc2138 reported cstate change: config changed from index 4 to 5, 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "da3584f429104d23a1a8899b35cc2138" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:36.037474 17766 raft_consensus.cc:2932] T b40f968655774ce887dff8b7f4651501 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 272b2179c1b4476090a89880df2a8baa (127.9.176.199) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } } }
I20240502 01:54:36.039733 20879 tablet_copy_client.cc:308] T f07e9b433e554c85b3c25b7e708f8b2f P 94931f4b51c9493cb3d4264ce99f5e76: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.195:37357
I20240502 01:54:36.041062 17789 tablet_copy_service.cc:140] P 84d3a4b174c34e02aedab1e620731359: Received BeginTabletCopySession request for tablet f07e9b433e554c85b3c25b7e708f8b2f from peer 94931f4b51c9493cb3d4264ce99f5e76 ({username='slave'} at 127.9.176.200:34957)
I20240502 01:54:36.041308 17789 tablet_copy_service.cc:161] P 84d3a4b174c34e02aedab1e620731359: Beginning new tablet copy session on tablet f07e9b433e554c85b3c25b7e708f8b2f from peer 94931f4b51c9493cb3d4264ce99f5e76 at {username='slave'} at 127.9.176.200:34957: session id = 94931f4b51c9493cb3d4264ce99f5e76-f07e9b433e554c85b3c25b7e708f8b2f
I20240502 01:54:36.043985 17631 raft_consensus.cc:2932] T 05b67d9912f047bfb6a2bed90b82ae15 P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
I20240502 01:54:36.047291 17789 tablet_copy_source_session.cc:205] T f07e9b433e554c85b3c25b7e708f8b2f P 84d3a4b174c34e02aedab1e620731359: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:36.056929 17632 consensus_queue.cc:237] T df16e178e04040699201f325ce9e2979 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } }
I20240502 01:54:36.060614 17765 raft_consensus.cc:1261] T df16e178e04040699201f325ce9e2979 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:36.061394 20635 consensus_queue.cc:1035] T df16e178e04040699201f325ce9e2979 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:36.062911 20783 tablet_copy_client.cc:308] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 94931f4b51c9493cb3d4264ce99f5e76: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.196:46381
I20240502 01:54:36.062690 17765 raft_consensus.cc:2932] T df16e178e04040699201f325ce9e2979 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 272b2179c1b4476090a89880df2a8baa (127.9.176.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } }
I20240502 01:54:36.066290 17922 tablet_copy_service.cc:140] P da3584f429104d23a1a8899b35cc2138: Received BeginTabletCopySession request for tablet 60ff2e50b1d74e0c8e0890b87f9ac8ba from peer 94931f4b51c9493cb3d4264ce99f5e76 ({username='slave'} at 127.9.176.200:36935)
I20240502 01:54:36.066509 17922 tablet_copy_service.cc:161] P da3584f429104d23a1a8899b35cc2138: Beginning new tablet copy session on tablet 60ff2e50b1d74e0c8e0890b87f9ac8ba from peer 94931f4b51c9493cb3d4264ce99f5e76 at {username='slave'} at 127.9.176.200:36935: session id = 94931f4b51c9493cb3d4264ce99f5e76-60ff2e50b1d74e0c8e0890b87f9ac8ba
I20240502 01:54:36.070082 17922 tablet_copy_source_session.cc:205] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P da3584f429104d23a1a8899b35cc2138: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:36.072326 20783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60ff2e50b1d74e0c8e0890b87f9ac8ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:36.080245 20790 tablet_bootstrap.cc:492] T 88a137bcfc76410583f910b377d1be7f P 4b7c382c9edd41babdd1a1afc904a813: Bootstrap starting.
I20240502 01:54:36.060097 19735 raft_consensus.cc:1261] T df16e178e04040699201f325ce9e2979 P 272b2179c1b4476090a89880df2a8baa [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:36.081991 20635 consensus_queue.cc:1035] T df16e178e04040699201f325ce9e2979 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:36.082396 19736 raft_consensus.cc:2932] T 2c78951695284b1aaab411c0a041c28a P 272b2179c1b4476090a89880df2a8baa [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 272b2179c1b4476090a89880df2a8baa (127.9.176.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } }
I20240502 01:54:36.084358 21143 consensus_queue.cc:1035] T 768e7cbf63a94d3a815ac77a4d0acf41 P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:36.087215 20293 raft_consensus.cc:2932] T 768e7cbf63a94d3a815ac77a4d0acf41 P 84d3a4b174c34e02aedab1e620731359 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 1b50a031f2784ba498b3e52e16916205 (127.9.176.194) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
I20240502 01:54:36.088207 19735 raft_consensus.cc:2932] T 768e7cbf63a94d3a815ac77a4d0acf41 P 272b2179c1b4476090a89880df2a8baa [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 1b50a031f2784ba498b3e52e16916205 (127.9.176.194) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
I20240502 01:54:36.023167 17898 raft_consensus.cc:2932] T 768e7cbf63a94d3a815ac77a4d0acf41 P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
I20240502 01:54:36.113313 21186 ts_tablet_manager.cc:902] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 272b2179c1b4476090a89880df2a8baa: Initiating tablet copy from peer da3584f429104d23a1a8899b35cc2138 (127.9.176.196:46381)
I20240502 01:54:36.112897 17767 raft_consensus.cc:2932] T 38a60d7db47149ff90444e5276fc8935 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 272b2179c1b4476090a89880df2a8baa (127.9.176.199) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } } }
I20240502 01:54:36.115343 17901 consensus_queue.cc:237] T 05b67d9912f047bfb6a2bed90b82ae15 P da3584f429104d23a1a8899b35cc2138 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } }
I20240502 01:54:36.115708 20879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f07e9b433e554c85b3c25b7e708f8b2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:36.119552 17762 raft_consensus.cc:1261] T 05b67d9912f047bfb6a2bed90b82ae15 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Refusing update from remote peer da3584f429104d23a1a8899b35cc2138: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:36.120518 21053 consensus_queue.cc:1035] T 05b67d9912f047bfb6a2bed90b82ae15 P da3584f429104d23a1a8899b35cc2138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:36.123178 21053 raft_consensus.cc:2932] T 05b67d9912f047bfb6a2bed90b82ae15 P da3584f429104d23a1a8899b35cc2138 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1b50a031f2784ba498b3e52e16916205 (127.9.176.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
I20240502 01:54:36.124503 17632 raft_consensus.cc:2932] T 2045e08a200148c69326351f38fd484d P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 29806d3d37354c169484f6db368f8798 (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } }
I20240502 01:54:36.126636 19467 raft_consensus.cc:1203] T 974f67157aa04d3e8b49b264cfa104e8 P 29806d3d37354c169484f6db368f8798 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 01:54:36.128607 17762 raft_consensus.cc:2932] T 05b67d9912f047bfb6a2bed90b82ae15 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1b50a031f2784ba498b3e52e16916205 (127.9.176.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
I20240502 01:54:36.130430 17340 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 05b67d9912f047bfb6a2bed90b82ae15 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:36.132741 17763 raft_consensus.cc:2932] T 2c78951695284b1aaab411c0a041c28a P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 272b2179c1b4476090a89880df2a8baa (127.9.176.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } }
I20240502 01:54:36.137990 20911 raft_consensus.cc:2932] T df16e178e04040699201f325ce9e2979 P 1b50a031f2784ba498b3e52e16916205 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } }
I20240502 01:54:36.138888 19446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:36.139842 17355 catalog_manager.cc:5484] T 05b67d9912f047bfb6a2bed90b82ae15 P da3584f429104d23a1a8899b35cc2138 reported cstate change: config changed from index 5 to 6, VOTER 1b50a031f2784ba498b3e52e16916205 (127.9.176.194) evicted. New cstate: current_term: 1 leader_uuid: "da3584f429104d23a1a8899b35cc2138" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:36.146324 17897 raft_consensus.cc:1261] T 768e7cbf63a94d3a815ac77a4d0acf41 P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:36.068187 21176 ts_tablet_manager.cc:902] T 38a60d7db47149ff90444e5276fc8935 P 272b2179c1b4476090a89880df2a8baa: Initiating tablet copy from peer 1b50a031f2784ba498b3e52e16916205 (127.9.176.194:45617)
I20240502 01:54:36.156294 17632 consensus_queue.cc:237] T 2c78951695284b1aaab411c0a041c28a P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } }
I20240502 01:54:36.155817 19737 raft_consensus.cc:2932] T df16e178e04040699201f325ce9e2979 P 272b2179c1b4476090a89880df2a8baa [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } }
I20240502 01:54:36.157389 17897 raft_consensus.cc:2932] T 2045e08a200148c69326351f38fd484d P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 29806d3d37354c169484f6db368f8798 (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } }
I20240502 01:54:36.119639 19604 raft_consensus.cc:1261] T 05b67d9912f047bfb6a2bed90b82ae15 P 4b7c382c9edd41babdd1a1afc904a813 [term 1 FOLLOWER]: Refusing update from remote peer da3584f429104d23a1a8899b35cc2138: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:36.160838 17498 raft_consensus.cc:1261] T 2c78951695284b1aaab411c0a041c28a P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240502 01:54:36.161015 17566 consensus_peers.cc:485] T 2c78951695284b1aaab411c0a041c28a P 1b50a031f2784ba498b3e52e16916205 -> Peer 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200:37831): Couldn't send request to peer 94931f4b51c9493cb3d4264ce99f5e76. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2c78951695284b1aaab411c0a041c28a. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:36.162159 19605 raft_consensus.cc:2932] T 05b67d9912f047bfb6a2bed90b82ae15 P 4b7c382c9edd41babdd1a1afc904a813 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
I20240502 01:54:36.162763 20635 consensus_queue.cc:1035] T 2c78951695284b1aaab411c0a041c28a P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:36.163659 20937 consensus_queue.cc:1035] T 05b67d9912f047bfb6a2bed90b82ae15 P da3584f429104d23a1a8899b35cc2138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:36.147393 17343 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet df16e178e04040699201f325ce9e2979 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20240502 01:54:36.168098 17432 consensus_peers.cc:485] T 37eda67598fe491bbb29aa62b2c4bec7 P 311f765f59754bdcb4eadc04e14bd4ea -> Peer 29806d3d37354c169484f6db368f8798 (127.9.176.197:35011): Couldn't send request to peer 29806d3d37354c169484f6db368f8798. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 37eda67598fe491bbb29aa62b2c4bec7. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:36.176457 21024 consensus_queue.cc:1035] T 768e7cbf63a94d3a815ac77a4d0acf41 P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:36.180027 19603 raft_consensus.cc:2932] T b40f968655774ce887dff8b7f4651501 P 4b7c382c9edd41babdd1a1afc904a813 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 272b2179c1b4476090a89880df2a8baa (127.9.176.199) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } } }
I20240502 01:54:36.194028 17897 raft_consensus.cc:2932] T 768e7cbf63a94d3a815ac77a4d0acf41 P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 1b50a031f2784ba498b3e52e16916205 (127.9.176.194) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
I20240502 01:54:36.196038 19738 raft_consensus.cc:1261] T 2c78951695284b1aaab411c0a041c28a P 272b2179c1b4476090a89880df2a8baa [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:36.197249 20911 consensus_queue.cc:1035] T 2c78951695284b1aaab411c0a041c28a P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:36.200650 20635 raft_consensus.cc:2932] T 2c78951695284b1aaab411c0a041c28a P 1b50a031f2784ba498b3e52e16916205 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } } }
I20240502 01:54:36.205930 17498 raft_consensus.cc:2932] T 2c78951695284b1aaab411c0a041c28a P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } } }
I20240502 01:54:36.211277 17761 consensus_queue.cc:237] T 2045e08a200148c69326351f38fd484d P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } }
I20240502 01:54:36.212885 19738 raft_consensus.cc:2932] T 2c78951695284b1aaab411c0a041c28a P 272b2179c1b4476090a89880df2a8baa [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } } }
I20240502 01:54:36.214017 17632 raft_consensus.cc:1261] T 2045e08a200148c69326351f38fd484d P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:36.214780 21006 consensus_queue.cc:1035] T 2045e08a200148c69326351f38fd484d P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:36.216730 17351 catalog_manager.cc:5484] T 768e7cbf63a94d3a815ac77a4d0acf41 P 272b2179c1b4476090a89880df2a8baa reported cstate change: config changed from index 7 to 8, VOTER 1b50a031f2784ba498b3e52e16916205 (127.9.176.194) evicted. New cstate: current_term: 1 leader_uuid: "84d3a4b174c34e02aedab1e620731359" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
I20240502 01:54:36.217645 17351 catalog_manager.cc:5484] T df16e178e04040699201f325ce9e2979 P 272b2179c1b4476090a89880df2a8baa reported cstate change: config changed from index 5 to 6, VOTER 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193) evicted. New cstate: current_term: 1 leader_uuid: "1b50a031f2784ba498b3e52e16916205" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } }
I20240502 01:54:36.226466 17613 tablet_service.cc:1514] Processing DeleteTablet for tablet 768e7cbf63a94d3a815ac77a4d0acf41 with delete_type TABLET_DATA_TOMBSTONED (TS 1b50a031f2784ba498b3e52e16916205 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:41484
I20240502 01:54:36.227957 17612 tablet_service.cc:1514] Processing DeleteTablet for tablet 05b67d9912f047bfb6a2bed90b82ae15 with delete_type TABLET_DATA_TOMBSTONED (TS 1b50a031f2784ba498b3e52e16916205 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41484
I20240502 01:54:36.233048 17351 catalog_manager.cc:5484] T 2c78951695284b1aaab411c0a041c28a P 272b2179c1b4476090a89880df2a8baa reported cstate change: config changed from index 5 to 6, NON_VOTER 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) added. New cstate: current_term: 1 leader_uuid: "1b50a031f2784ba498b3e52e16916205" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } } }
I20240502 01:54:36.235735 20783 tablet_copy_client.cc:783] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 94931f4b51c9493cb3d4264ce99f5e76: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:36.236074 20783 tablet_copy_client.cc:647] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 94931f4b51c9493cb3d4264ce99f5e76: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:36.238348 20783 tablet_copy_client.cc:523] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 94931f4b51c9493cb3d4264ce99f5e76: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:36.240638 19604 raft_consensus.cc:2932] T 05b67d9912f047bfb6a2bed90b82ae15 P 4b7c382c9edd41babdd1a1afc904a813 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1b50a031f2784ba498b3e52e16916205 (127.9.176.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
I20240502 01:54:36.244097 21200 tablet_replica.cc:330] T 768e7cbf63a94d3a815ac77a4d0acf41 P 1b50a031f2784ba498b3e52e16916205: stopping tablet replica
I20240502 01:54:36.244661 21200 raft_consensus.cc:2229] T 768e7cbf63a94d3a815ac77a4d0acf41 P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:36.245226 21200 raft_consensus.cc:2258] T 768e7cbf63a94d3a815ac77a4d0acf41 P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:36.245071 21143 raft_consensus.cc:2932] T 2045e08a200148c69326351f38fd484d P 84d3a4b174c34e02aedab1e620731359 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER da3584f429104d23a1a8899b35cc2138 (127.9.176.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } }
I20240502 01:54:36.256305 21200 ts_tablet_manager.cc:1875] T 768e7cbf63a94d3a815ac77a4d0acf41 P 1b50a031f2784ba498b3e52e16916205: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:36.259688 20879 tablet_copy_client.cc:783] T f07e9b433e554c85b3c25b7e708f8b2f P 94931f4b51c9493cb3d4264ce99f5e76: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:36.260015 20879 tablet_copy_client.cc:647] T f07e9b433e554c85b3c25b7e708f8b2f P 94931f4b51c9493cb3d4264ce99f5e76: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:36.268787 17880 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:54:36.269560 20783 tablet_bootstrap.cc:492] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 94931f4b51c9493cb3d4264ce99f5e76: Bootstrap starting.
I20240502 01:54:36.269930 19871 raft_consensus.cc:2932] T 38a60d7db47149ff90444e5276fc8935 P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 272b2179c1b4476090a89880df2a8baa (127.9.176.199) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } } }
I20240502 01:54:36.277137 17477 tablet_service.cc:1514] Processing DeleteTablet for tablet df16e178e04040699201f325ce9e2979 with delete_type TABLET_DATA_TOMBSTONED (TS 311f765f59754bdcb4eadc04e14bd4ea not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:48974
I20240502 01:54:36.290354 21207 tablet_replica.cc:330] T df16e178e04040699201f325ce9e2979 P 311f765f59754bdcb4eadc04e14bd4ea: stopping tablet replica
I20240502 01:54:36.291095 21207 raft_consensus.cc:2229] T df16e178e04040699201f325ce9e2979 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:36.293193 21207 raft_consensus.cc:2258] T df16e178e04040699201f325ce9e2979 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:36.293814 17632 raft_consensus.cc:2932] T 2045e08a200148c69326351f38fd484d P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER da3584f429104d23a1a8899b35cc2138 (127.9.176.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } }
I20240502 01:54:36.305003 19585 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:36.312060 21186 tablet_copy_client.cc:308] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 272b2179c1b4476090a89880df2a8baa: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.196:46381
I20240502 01:54:36.332726 21212 ts_tablet_manager.cc:902] T 37eda67598fe491bbb29aa62b2c4bec7 P 29806d3d37354c169484f6db368f8798: Initiating tablet copy from peer 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193:35397)
I20240502 01:54:36.345111 21207 ts_tablet_manager.cc:1875] T df16e178e04040699201f325ce9e2979 P 311f765f59754bdcb4eadc04e14bd4ea: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:36.347503 17897 consensus_queue.cc:237] T 17365abbb6c1430aa278719eb8e4a648 P da3584f429104d23a1a8899b35cc2138 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } }
I20240502 01:54:36.346132 17351 catalog_manager.cc:5484] T 2045e08a200148c69326351f38fd484d P 1b50a031f2784ba498b3e52e16916205 reported cstate change: config changed from index 5 to 6, VOTER da3584f429104d23a1a8899b35cc2138 (127.9.176.196) evicted. New cstate: current_term: 1 leader_uuid: "84d3a4b174c34e02aedab1e620731359" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } }
I20240502 01:54:36.354326 17880 tablet_service.cc:1514] Processing DeleteTablet for tablet 2045e08a200148c69326351f38fd484d with delete_type TABLET_DATA_TOMBSTONED (TS da3584f429104d23a1a8899b35cc2138 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:34396
I20240502 01:54:36.374670 21220 tablet_replica.cc:330] T 2045e08a200148c69326351f38fd484d P da3584f429104d23a1a8899b35cc2138: stopping tablet replica
I20240502 01:54:36.376735 19467 raft_consensus.cc:1261] T 2045e08a200148c69326351f38fd484d P 29806d3d37354c169484f6db368f8798 [term 1 FOLLOWER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:36.377084 17763 raft_consensus.cc:1261] T 17365abbb6c1430aa278719eb8e4a648 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Refusing update from remote peer da3584f429104d23a1a8899b35cc2138: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:36.377584 21200 ts_tablet_manager.cc:1888] T 768e7cbf63a94d3a815ac77a4d0acf41 P 1b50a031f2784ba498b3e52e16916205: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240502 01:54:36.377852 21200 log.cc:1196] T 768e7cbf63a94d3a815ac77a4d0acf41 P 1b50a031f2784ba498b3e52e16916205: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/wal/wals/768e7cbf63a94d3a815ac77a4d0acf41
I20240502 01:54:36.377794 20937 consensus_queue.cc:1035] T 17365abbb6c1430aa278719eb8e4a648 P da3584f429104d23a1a8899b35cc2138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:36.379132 21200 tablet_replica.cc:330] T 05b67d9912f047bfb6a2bed90b82ae15 P 1b50a031f2784ba498b3e52e16916205: stopping tablet replica
I20240502 01:54:36.379195 17343 catalog_manager.cc:4830] TS 1b50a031f2784ba498b3e52e16916205 (127.9.176.194:45617): tablet 768e7cbf63a94d3a815ac77a4d0acf41 (table rebalance_test_table_1 [id=952912d7ec8d493bbb8985fffe9aa918]) successfully deleted
I20240502 01:54:36.379760 17767 raft_consensus.cc:1261] T 2c78951695284b1aaab411c0a041c28a P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:36.380760 17343 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 768e7cbf63a94d3a815ac77a4d0acf41 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:36.381297 17343 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2045e08a200148c69326351f38fd484d with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:36.382377 19852 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:36.381496 21220 raft_consensus.cc:2229] T 2045e08a200148c69326351f38fd484d P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:36.383798 21220 raft_consensus.cc:2258] T 2045e08a200148c69326351f38fd484d P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:36.385908 17922 tablet_copy_service.cc:140] P da3584f429104d23a1a8899b35cc2138: Received BeginTabletCopySession request for tablet 60ff2e50b1d74e0c8e0890b87f9ac8ba from peer 272b2179c1b4476090a89880df2a8baa ({username='slave'} at 127.9.176.199:46107)
I20240502 01:54:36.386119 17922 tablet_copy_service.cc:161] P da3584f429104d23a1a8899b35cc2138: Beginning new tablet copy session on tablet 60ff2e50b1d74e0c8e0890b87f9ac8ba from peer 272b2179c1b4476090a89880df2a8baa at {username='slave'} at 127.9.176.199:46107: session id = 272b2179c1b4476090a89880df2a8baa-60ff2e50b1d74e0c8e0890b87f9ac8ba
I20240502 01:54:36.386163 21220 ts_tablet_manager.cc:1875] T 2045e08a200148c69326351f38fd484d P da3584f429104d23a1a8899b35cc2138: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240502 01:54:36.388558 17832 consensus_peers.cc:485] T 17365abbb6c1430aa278719eb8e4a648 P da3584f429104d23a1a8899b35cc2138 -> Peer 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198:36865): Couldn't send request to peer 4b7c382c9edd41babdd1a1afc904a813. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 17365abbb6c1430aa278719eb8e4a648. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:36.389006 19605 raft_consensus.cc:2932] T 768e7cbf63a94d3a815ac77a4d0acf41 P 4b7c382c9edd41babdd1a1afc904a813 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 1b50a031f2784ba498b3e52e16916205 (127.9.176.194) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
I20240502 01:54:36.389482 17922 tablet_copy_source_session.cc:205] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P da3584f429104d23a1a8899b35cc2138: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:36.392524 21186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60ff2e50b1d74e0c8e0890b87f9ac8ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:36.407678 21186 tablet_copy_client.cc:783] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 272b2179c1b4476090a89880df2a8baa: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:36.408376 21186 tablet_copy_client.cc:647] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 272b2179c1b4476090a89880df2a8baa: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:36.380116 17631 raft_consensus.cc:1261] T 17365abbb6c1430aa278719eb8e4a648 P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Refusing update from remote peer da3584f429104d23a1a8899b35cc2138: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:36.410843 21200 raft_consensus.cc:2229] T 05b67d9912f047bfb6a2bed90b82ae15 P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:36.411221 21200 raft_consensus.cc:2258] T 05b67d9912f047bfb6a2bed90b82ae15 P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:36.412158 20937 consensus_queue.cc:1035] T 17365abbb6c1430aa278719eb8e4a648 P da3584f429104d23a1a8899b35cc2138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:36.412885 21200 ts_tablet_manager.cc:1875] T 05b67d9912f047bfb6a2bed90b82ae15 P 1b50a031f2784ba498b3e52e16916205: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:36.414708 21186 tablet_copy_client.cc:523] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 272b2179c1b4476090a89880df2a8baa: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:36.415695 21053 raft_consensus.cc:2932] T 17365abbb6c1430aa278719eb8e4a648 P da3584f429104d23a1a8899b35cc2138 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } } }
I20240502 01:54:36.422228 17350 catalog_manager.cc:5484] T 17365abbb6c1430aa278719eb8e4a648 P da3584f429104d23a1a8899b35cc2138 reported cstate change: config changed from index -1 to 4, NON_VOTER 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) added. New cstate: current_term: 1 leader_uuid: "da3584f429104d23a1a8899b35cc2138" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:36.419143 20628 consensus_queue.cc:1035] T 2c78951695284b1aaab411c0a041c28a P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:36.425233 20879 tablet_copy_client.cc:523] T f07e9b433e554c85b3c25b7e708f8b2f P 94931f4b51c9493cb3d4264ce99f5e76: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:36.428864 20879 tablet_bootstrap.cc:492] T f07e9b433e554c85b3c25b7e708f8b2f P 94931f4b51c9493cb3d4264ce99f5e76: Bootstrap starting.
I20240502 01:54:36.431952 21006 consensus_queue.cc:1035] T 2045e08a200148c69326351f38fd484d P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:36.433091 17763 raft_consensus.cc:2932] T 17365abbb6c1430aa278719eb8e4a648 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } } }
I20240502 01:54:36.437716 17767 raft_consensus.cc:2932] T df16e178e04040699201f325ce9e2979 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } }
I20240502 01:54:36.442965 17764 raft_consensus.cc:2932] T 2c78951695284b1aaab411c0a041c28a P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } } }
I20240502 01:54:36.444909 21186 tablet_bootstrap.cc:492] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 272b2179c1b4476090a89880df2a8baa: Bootstrap starting.
I20240502 01:54:36.449651 21207 ts_tablet_manager.cc:1888] T df16e178e04040699201f325ce9e2979 P 311f765f59754bdcb4eadc04e14bd4ea: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:54:36.455818 21176 tablet_copy_client.cc:308] T 38a60d7db47149ff90444e5276fc8935 P 272b2179c1b4476090a89880df2a8baa: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.194:45617
I20240502 01:54:36.456063 17631 raft_consensus.cc:2932] T 17365abbb6c1430aa278719eb8e4a648 P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } } }
I20240502 01:54:36.456312 19467 raft_consensus.cc:2932] T 2045e08a200148c69326351f38fd484d P 29806d3d37354c169484f6db368f8798 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER da3584f429104d23a1a8899b35cc2138 (127.9.176.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } }
I20240502 01:54:36.457028 17654 tablet_copy_service.cc:140] P 1b50a031f2784ba498b3e52e16916205: Received BeginTabletCopySession request for tablet 38a60d7db47149ff90444e5276fc8935 from peer 272b2179c1b4476090a89880df2a8baa ({username='slave'} at 127.9.176.199:56005)
I20240502 01:54:36.457197 17654 tablet_copy_service.cc:161] P 1b50a031f2784ba498b3e52e16916205: Beginning new tablet copy session on tablet 38a60d7db47149ff90444e5276fc8935 from peer 272b2179c1b4476090a89880df2a8baa at {username='slave'} at 127.9.176.199:56005: session id = 272b2179c1b4476090a89880df2a8baa-38a60d7db47149ff90444e5276fc8935
I20240502 01:54:36.460817 17654 tablet_copy_source_session.cc:205] T 38a60d7db47149ff90444e5276fc8935 P 1b50a031f2784ba498b3e52e16916205: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:36.462826 21176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38a60d7db47149ff90444e5276fc8935. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:36.470662 21207 log.cc:1196] T df16e178e04040699201f325ce9e2979 P 311f765f59754bdcb4eadc04e14bd4ea: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/wal/wals/df16e178e04040699201f325ce9e2979
I20240502 01:54:36.472304 17343 catalog_manager.cc:4830] TS 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193:35397): tablet df16e178e04040699201f325ce9e2979 (table rebalance_test_table_8 [id=40a6be2c1a2744428cea56b78225e44e]) successfully deleted
I20240502 01:54:36.477234 17497 consensus_queue.cc:237] T a6b5231a18a5414dbd010d4d48f08dbd P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } }
I20240502 01:54:36.485639 17746 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:54:36.489567 21212 tablet_copy_client.cc:308] T 37eda67598fe491bbb29aa62b2c4bec7 P 29806d3d37354c169484f6db368f8798: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.193:35397
I20240502 01:54:36.503315 17632 raft_consensus.cc:1261] T a6b5231a18a5414dbd010d4d48f08dbd P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Refusing update from remote peer 311f765f59754bdcb4eadc04e14bd4ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:36.504659 20788 consensus_queue.cc:1035] T a6b5231a18a5414dbd010d4d48f08dbd P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 01:54:36.507175 17432 consensus_peers.cc:485] T a6b5231a18a5414dbd010d4d48f08dbd P 311f765f59754bdcb4eadc04e14bd4ea -> Peer 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198:36865): Couldn't send request to peer 4b7c382c9edd41babdd1a1afc904a813. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a6b5231a18a5414dbd010d4d48f08dbd. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:36.513597 17897 raft_consensus.cc:1261] T a6b5231a18a5414dbd010d4d48f08dbd P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Refusing update from remote peer 311f765f59754bdcb4eadc04e14bd4ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:36.514318 20788 consensus_queue.cc:1035] T a6b5231a18a5414dbd010d4d48f08dbd P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:36.520331 17763 consensus_queue.cc:237] T 768e7cbf63a94d3a815ac77a4d0acf41 P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } }
I20240502 01:54:36.521057 21220 ts_tablet_manager.cc:1888] T 2045e08a200148c69326351f38fd484d P da3584f429104d23a1a8899b35cc2138: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:54:36.521225 21220 log.cc:1196] T 2045e08a200148c69326351f38fd484d P da3584f429104d23a1a8899b35cc2138: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/wal/wals/2045e08a200148c69326351f38fd484d
I20240502 01:54:36.524668 17340 catalog_manager.cc:4830] TS da3584f429104d23a1a8899b35cc2138 (127.9.176.196:46381): tablet 2045e08a200148c69326351f38fd484d (table rebalance_test_table_1 [id=952912d7ec8d493bbb8985fffe9aa918]) successfully deleted
I20240502 01:54:36.525835 17764 consensus_queue.cc:237] T 6da59c0626c64a5988e39a831c42c268 P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } }
I20240502 01:54:36.527350 17520 tablet_copy_service.cc:140] P 311f765f59754bdcb4eadc04e14bd4ea: Received BeginTabletCopySession request for tablet 37eda67598fe491bbb29aa62b2c4bec7 from peer 29806d3d37354c169484f6db368f8798 ({username='slave'} at 127.9.176.197:51011)
I20240502 01:54:36.527539 17520 tablet_copy_service.cc:161] P 311f765f59754bdcb4eadc04e14bd4ea: Beginning new tablet copy session on tablet 37eda67598fe491bbb29aa62b2c4bec7 from peer 29806d3d37354c169484f6db368f8798 at {username='slave'} at 127.9.176.197:51011: session id = 29806d3d37354c169484f6db368f8798-37eda67598fe491bbb29aa62b2c4bec7
I20240502 01:54:36.528033 19871 raft_consensus.cc:1261] T 6da59c0626c64a5988e39a831c42c268 P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:36.529747 21006 consensus_queue.cc:1035] T 6da59c0626c64a5988e39a831c42c268 P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:36.530422 17520 tablet_copy_source_session.cc:205] T 37eda67598fe491bbb29aa62b2c4bec7 P 311f765f59754bdcb4eadc04e14bd4ea: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:36.530227 20788 raft_consensus.cc:2932] T a6b5231a18a5414dbd010d4d48f08dbd P 311f765f59754bdcb4eadc04e14bd4ea [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } } }
I20240502 01:54:36.531637 19738 raft_consensus.cc:1261] T 768e7cbf63a94d3a815ac77a4d0acf41 P 272b2179c1b4476090a89880df2a8baa [term 1 FOLLOWER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:54:36.531993 19605 raft_consensus.cc:1261] T 768e7cbf63a94d3a815ac77a4d0acf41 P 4b7c382c9edd41babdd1a1afc904a813 [term 1 FOLLOWER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:54:36.532665 21007 consensus_queue.cc:1035] T 768e7cbf63a94d3a815ac77a4d0acf41 P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:36.533629 21007 consensus_queue.cc:1035] T 768e7cbf63a94d3a815ac77a4d0acf41 P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:36.533749 19467 raft_consensus.cc:1261] T 6da59c0626c64a5988e39a831c42c268 P 29806d3d37354c169484f6db368f8798 [term 1 FOLLOWER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:36.536906 21212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37eda67598fe491bbb29aa62b2c4bec7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:36.538801 17898 raft_consensus.cc:2932] T a6b5231a18a5414dbd010d4d48f08dbd P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } } }
I20240502 01:54:36.539088 17355 catalog_manager.cc:5484] T a6b5231a18a5414dbd010d4d48f08dbd P 311f765f59754bdcb4eadc04e14bd4ea reported cstate change: config changed from index -1 to 4, NON_VOTER 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) added. New cstate: current_term: 1 leader_uuid: "311f765f59754bdcb4eadc04e14bd4ea" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:36.537220 17632 raft_consensus.cc:2932] T a6b5231a18a5414dbd010d4d48f08dbd P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } } }
I20240502 01:54:36.545512 21212 tablet_copy_client.cc:783] T 37eda67598fe491bbb29aa62b2c4bec7 P 29806d3d37354c169484f6db368f8798: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:36.548179 17897 raft_consensus.cc:1261] T 6da59c0626c64a5988e39a831c42c268 P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:36.548686 21212 tablet_copy_client.cc:647] T 37eda67598fe491bbb29aa62b2c4bec7 P 29806d3d37354c169484f6db368f8798: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:36.556989 20911 raft_consensus.cc:1050] T 38a60d7db47149ff90444e5276fc8935 P 1b50a031f2784ba498b3e52e16916205: attempting to promote NON_VOTER 94931f4b51c9493cb3d4264ce99f5e76 to VOTER
I20240502 01:54:36.558147 20911 consensus_queue.cc:237] T 38a60d7db47149ff90444e5276fc8935 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } }
I20240502 01:54:36.564258 21212 tablet_copy_client.cc:523] T 37eda67598fe491bbb29aa62b2c4bec7 P 29806d3d37354c169484f6db368f8798: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:36.566097 21007 consensus_queue.cc:1035] T 6da59c0626c64a5988e39a831c42c268 P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:36.567335 21007 consensus_queue.cc:1035] T 6da59c0626c64a5988e39a831c42c268 P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:36.569185 21212 tablet_bootstrap.cc:492] T 37eda67598fe491bbb29aa62b2c4bec7 P 29806d3d37354c169484f6db368f8798: Bootstrap starting.
I20240502 01:54:36.569478 21006 raft_consensus.cc:2932] T 768e7cbf63a94d3a815ac77a4d0acf41 P 84d3a4b174c34e02aedab1e620731359 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER da3584f429104d23a1a8899b35cc2138 (127.9.176.196) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
I20240502 01:54:36.572485 20628 raft_consensus.cc:1050] T 974f67157aa04d3e8b49b264cfa104e8 P 1b50a031f2784ba498b3e52e16916205: attempting to promote NON_VOTER 29806d3d37354c169484f6db368f8798 to VOTER
I20240502 01:54:36.573338 20628 consensus_queue.cc:237] T 974f67157aa04d3e8b49b264cfa104e8 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } }
I20240502 01:54:36.575729 17343 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 768e7cbf63a94d3a815ac77a4d0acf41 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:36.576671 21176 tablet_copy_client.cc:783] T 38a60d7db47149ff90444e5276fc8935 P 272b2179c1b4476090a89880df2a8baa: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:36.577155 21200 ts_tablet_manager.cc:1888] T 05b67d9912f047bfb6a2bed90b82ae15 P 1b50a031f2784ba498b3e52e16916205: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:54:36.577495 17354 catalog_manager.cc:5484] T 768e7cbf63a94d3a815ac77a4d0acf41 P 84d3a4b174c34e02aedab1e620731359 reported cstate change: config changed from index 8 to 9, VOTER da3584f429104d23a1a8899b35cc2138 (127.9.176.196) evicted. New cstate: current_term: 1 leader_uuid: "84d3a4b174c34e02aedab1e620731359" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:36.579946 21176 tablet_copy_client.cc:647] T 38a60d7db47149ff90444e5276fc8935 P 272b2179c1b4476090a89880df2a8baa: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:36.584190 21200 log.cc:1196] T 05b67d9912f047bfb6a2bed90b82ae15 P 1b50a031f2784ba498b3e52e16916205: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/wal/wals/05b67d9912f047bfb6a2bed90b82ae15
I20240502 01:54:36.586453 17343 catalog_manager.cc:4830] TS 1b50a031f2784ba498b3e52e16916205 (127.9.176.194:45617): tablet 05b67d9912f047bfb6a2bed90b82ae15 (table rebalance_test_table_9 [id=60aa7e7a81564c3a888cc9b68700c9ad]) successfully deleted
I20240502 01:54:36.599647 21176 tablet_copy_client.cc:523] T 38a60d7db47149ff90444e5276fc8935 P 272b2179c1b4476090a89880df2a8baa: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:36.601017 19872 raft_consensus.cc:1261] T 38a60d7db47149ff90444e5276fc8935 P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 LEARNER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:36.600893 19605 raft_consensus.cc:2932] T 768e7cbf63a94d3a815ac77a4d0acf41 P 4b7c382c9edd41babdd1a1afc904a813 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER da3584f429104d23a1a8899b35cc2138 (127.9.176.196) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
I20240502 01:54:36.601514 19738 raft_consensus.cc:2932] T 768e7cbf63a94d3a815ac77a4d0acf41 P 272b2179c1b4476090a89880df2a8baa [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER da3584f429104d23a1a8899b35cc2138 (127.9.176.196) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
I20240502 01:54:36.606070 17880 tablet_service.cc:1514] Processing DeleteTablet for tablet 768e7cbf63a94d3a815ac77a4d0acf41 with delete_type TABLET_DATA_TOMBSTONED (TS da3584f429104d23a1a8899b35cc2138 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:34396
I20240502 01:54:36.607442 17764 raft_consensus.cc:1261] T 974f67157aa04d3e8b49b264cfa104e8 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:36.607800 21220 tablet_replica.cc:330] T 768e7cbf63a94d3a815ac77a4d0acf41 P da3584f429104d23a1a8899b35cc2138: stopping tablet replica
I20240502 01:54:36.608063 21220 raft_consensus.cc:2229] T 768e7cbf63a94d3a815ac77a4d0acf41 P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:36.608579 21220 raft_consensus.cc:2258] T 768e7cbf63a94d3a815ac77a4d0acf41 P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:36.609083 21007 raft_consensus.cc:2932] T 6da59c0626c64a5988e39a831c42c268 P 84d3a4b174c34e02aedab1e620731359 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
I20240502 01:54:36.609824 21220 ts_tablet_manager.cc:1875] T 768e7cbf63a94d3a815ac77a4d0acf41 P da3584f429104d23a1a8899b35cc2138: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:36.613052 20920 consensus_queue.cc:1035] T 38a60d7db47149ff90444e5276fc8935 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.009s
I20240502 01:54:36.614400 19467 raft_consensus.cc:1261] T 974f67157aa04d3e8b49b264cfa104e8 P 29806d3d37354c169484f6db368f8798 [term 1 LEARNER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:36.615159 20920 consensus_queue.cc:1035] T 974f67157aa04d3e8b49b264cfa104e8 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:36.618052 19872 raft_consensus.cc:2932] T 6da59c0626c64a5988e39a831c42c268 P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
I20240502 01:54:36.624751 20920 consensus_queue.cc:1035] T 974f67157aa04d3e8b49b264cfa104e8 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:36.625646 17497 raft_consensus.cc:1261] T 38a60d7db47149ff90444e5276fc8935 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:36.626726 20913 consensus_queue.cc:1035] T 38a60d7db47149ff90444e5276fc8935 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:36.631911 17897 raft_consensus.cc:1261] T 974f67157aa04d3e8b49b264cfa104e8 P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:36.633241 21242 consensus_queue.cc:1035] T 974f67157aa04d3e8b49b264cfa104e8 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:36.635329 21220 ts_tablet_manager.cc:1888] T 768e7cbf63a94d3a815ac77a4d0acf41 P da3584f429104d23a1a8899b35cc2138: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 01:54:36.639456 21220 log.cc:1196] T 768e7cbf63a94d3a815ac77a4d0acf41 P da3584f429104d23a1a8899b35cc2138: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/wal/wals/768e7cbf63a94d3a815ac77a4d0acf41
I20240502 01:54:36.642566 21241 ts_tablet_manager.cc:902] T 2c78951695284b1aaab411c0a041c28a P 94931f4b51c9493cb3d4264ce99f5e76: Initiating tablet copy from peer 1b50a031f2784ba498b3e52e16916205 (127.9.176.194:45617)
I20240502 01:54:36.639600 17898 raft_consensus.cc:2932] T 6da59c0626c64a5988e39a831c42c268 P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
I20240502 01:54:36.654560 21176 tablet_bootstrap.cc:492] T 38a60d7db47149ff90444e5276fc8935 P 272b2179c1b4476090a89880df2a8baa: Bootstrap starting.
W20240502 01:54:36.655050 17565 consensus_peers.cc:485] T 38a60d7db47149ff90444e5276fc8935 P 1b50a031f2784ba498b3e52e16916205 -> Peer 272b2179c1b4476090a89880df2a8baa (127.9.176.199:42269): Couldn't send request to peer 272b2179c1b4476090a89880df2a8baa. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:36.657310 17343 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6da59c0626c64a5988e39a831c42c268 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:36.658098 17340 catalog_manager.cc:4830] TS da3584f429104d23a1a8899b35cc2138 (127.9.176.196:46381): tablet 768e7cbf63a94d3a815ac77a4d0acf41 (table rebalance_test_table_1 [id=952912d7ec8d493bbb8985fffe9aa918]) successfully deleted
I20240502 01:54:36.664587 20911 raft_consensus.cc:2932] T 974f67157aa04d3e8b49b264cfa104e8 P 1b50a031f2784ba498b3e52e16916205 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 29806d3d37354c169484f6db368f8798 (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } }
I20240502 01:54:36.666714 17631 consensus_queue.cc:237] T b40f968655774ce887dff8b7f4651501 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } }
I20240502 01:54:36.668531 17897 raft_consensus.cc:2932] T 974f67157aa04d3e8b49b264cfa104e8 P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 29806d3d37354c169484f6db368f8798 (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } }
I20240502 01:54:36.671972 19467 raft_consensus.cc:2932] T 974f67157aa04d3e8b49b264cfa104e8 P 29806d3d37354c169484f6db368f8798 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 29806d3d37354c169484f6db368f8798 (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } }
I20240502 01:54:36.674682 17348 catalog_manager.cc:5484] T 6da59c0626c64a5988e39a831c42c268 P 84d3a4b174c34e02aedab1e620731359 reported cstate change: config changed from index 7 to 8, VOTER 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193) evicted. New cstate: current_term: 1 leader_uuid: "84d3a4b174c34e02aedab1e620731359" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:36.681502 17351 catalog_manager.cc:5484] T 974f67157aa04d3e8b49b264cfa104e8 P 29806d3d37354c169484f6db368f8798 reported cstate change: config changed from index 4 to 5, 29806d3d37354c169484f6db368f8798 (127.9.176.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1b50a031f2784ba498b3e52e16916205" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } }
I20240502 01:54:36.684818 21241 tablet_copy_client.cc:308] T 2c78951695284b1aaab411c0a041c28a P 94931f4b51c9493cb3d4264ce99f5e76: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.194:45617
I20240502 01:54:36.685811 17654 tablet_copy_service.cc:140] P 1b50a031f2784ba498b3e52e16916205: Received BeginTabletCopySession request for tablet 2c78951695284b1aaab411c0a041c28a from peer 94931f4b51c9493cb3d4264ce99f5e76 ({username='slave'} at 127.9.176.200:37843)
I20240502 01:54:36.686028 17654 tablet_copy_service.cc:161] P 1b50a031f2784ba498b3e52e16916205: Beginning new tablet copy session on tablet 2c78951695284b1aaab411c0a041c28a from peer 94931f4b51c9493cb3d4264ce99f5e76 at {username='slave'} at 127.9.176.200:37843: session id = 94931f4b51c9493cb3d4264ce99f5e76-2c78951695284b1aaab411c0a041c28a
I20240502 01:54:36.689792 17654 tablet_copy_source_session.cc:205] T 2c78951695284b1aaab411c0a041c28a P 1b50a031f2784ba498b3e52e16916205: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:36.693503 21241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c78951695284b1aaab411c0a041c28a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:36.695216 17763 consensus_queue.cc:237] T 6da59c0626c64a5988e39a831c42c268 P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } }
I20240502 01:54:36.697739 19871 raft_consensus.cc:1261] T 6da59c0626c64a5988e39a831c42c268 P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:54:36.702157 21007 consensus_queue.cc:1035] T 6da59c0626c64a5988e39a831c42c268 P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:36.702359 17763 consensus_queue.cc:237] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } }
I20240502 01:54:36.704667 21024 raft_consensus.cc:2932] T 6da59c0626c64a5988e39a831c42c268 P 84d3a4b174c34e02aedab1e620731359 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER da3584f429104d23a1a8899b35cc2138 (127.9.176.196) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
I20240502 01:54:36.716878 17343 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6da59c0626c64a5988e39a831c42c268 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:36.720556 17355 catalog_manager.cc:5484] T 6da59c0626c64a5988e39a831c42c268 P 84d3a4b174c34e02aedab1e620731359 reported cstate change: config changed from index 8 to 9, VOTER da3584f429104d23a1a8899b35cc2138 (127.9.176.196) evicted. New cstate: current_term: 1 leader_uuid: "84d3a4b174c34e02aedab1e620731359" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:36.720829 17477 tablet_service.cc:1514] Processing DeleteTablet for tablet 6da59c0626c64a5988e39a831c42c268 with delete_type TABLET_DATA_TOMBSTONED (TS 311f765f59754bdcb4eadc04e14bd4ea not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:48974
I20240502 01:54:36.721196 21207 tablet_replica.cc:330] T 6da59c0626c64a5988e39a831c42c268 P 311f765f59754bdcb4eadc04e14bd4ea: stopping tablet replica
I20240502 01:54:36.721443 21207 raft_consensus.cc:2229] T 6da59c0626c64a5988e39a831c42c268 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:36.721836 21207 raft_consensus.cc:2258] T 6da59c0626c64a5988e39a831c42c268 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:36.722240 17499 raft_consensus.cc:1261] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:36.721885 19871 raft_consensus.cc:2932] T 6da59c0626c64a5988e39a831c42c268 P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER da3584f429104d23a1a8899b35cc2138 (127.9.176.196) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
I20240502 01:54:36.723405 21007 consensus_queue.cc:1035] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:36.723862 21207 ts_tablet_manager.cc:1875] T 6da59c0626c64a5988e39a831c42c268 P 311f765f59754bdcb4eadc04e14bd4ea: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:36.726006 17880 tablet_service.cc:1514] Processing DeleteTablet for tablet 6da59c0626c64a5988e39a831c42c268 with delete_type TABLET_DATA_TOMBSTONED (TS da3584f429104d23a1a8899b35cc2138 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:34396
I20240502 01:54:36.726378 21220 tablet_replica.cc:330] T 6da59c0626c64a5988e39a831c42c268 P da3584f429104d23a1a8899b35cc2138: stopping tablet replica
I20240502 01:54:36.726593 21220 raft_consensus.cc:2229] T 6da59c0626c64a5988e39a831c42c268 P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:36.727067 21220 raft_consensus.cc:2258] T 6da59c0626c64a5988e39a831c42c268 P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:36.727381 21241 tablet_copy_client.cc:783] T 2c78951695284b1aaab411c0a041c28a P 94931f4b51c9493cb3d4264ce99f5e76: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:36.728525 21220 ts_tablet_manager.cc:1875] T 6da59c0626c64a5988e39a831c42c268 P da3584f429104d23a1a8899b35cc2138: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:36.730876 21207 ts_tablet_manager.cc:1888] T 6da59c0626c64a5988e39a831c42c268 P 311f765f59754bdcb4eadc04e14bd4ea: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240502 01:54:36.731367 21207 log.cc:1196] T 6da59c0626c64a5988e39a831c42c268 P 311f765f59754bdcb4eadc04e14bd4ea: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/wal/wals/6da59c0626c64a5988e39a831c42c268
I20240502 01:54:36.733024 17343 catalog_manager.cc:4830] TS 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193:35397): tablet 6da59c0626c64a5988e39a831c42c268 (table rebalance_test_table_0 [id=c516e7439e734f02a0947c3680cf79e2]) successfully deleted
I20240502 01:54:36.734476 20879 tablet_bootstrap.cc:492] T f07e9b433e554c85b3c25b7e708f8b2f P 94931f4b51c9493cb3d4264ce99f5e76: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:36.745489 21220 ts_tablet_manager.cc:1888] T 6da59c0626c64a5988e39a831c42c268 P da3584f429104d23a1a8899b35cc2138: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 01:54:36.745920 21220 log.cc:1196] T 6da59c0626c64a5988e39a831c42c268 P da3584f429104d23a1a8899b35cc2138: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/wal/wals/6da59c0626c64a5988e39a831c42c268
I20240502 01:54:36.747027 17340 catalog_manager.cc:4830] TS da3584f429104d23a1a8899b35cc2138 (127.9.176.196:46381): tablet 6da59c0626c64a5988e39a831c42c268 (table rebalance_test_table_0 [id=c516e7439e734f02a0947c3680cf79e2]) successfully deleted
I20240502 01:54:36.747669 20879 tablet_bootstrap.cc:492] T f07e9b433e554c85b3c25b7e708f8b2f P 94931f4b51c9493cb3d4264ce99f5e76: Bootstrap complete.
I20240502 01:54:36.748016 20879 ts_tablet_manager.cc:1367] T f07e9b433e554c85b3c25b7e708f8b2f P 94931f4b51c9493cb3d4264ce99f5e76: Time spent bootstrapping tablet: real 0.319s	user 0.055s	sys 0.000s
I20240502 01:54:36.748886 20879 raft_consensus.cc:348] T f07e9b433e554c85b3c25b7e708f8b2f P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } }
I20240502 01:54:36.749199 20879 raft_consensus.cc:729] T f07e9b433e554c85b3c25b7e708f8b2f P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 94931f4b51c9493cb3d4264ce99f5e76, State: Initialized, Role: LEARNER
I20240502 01:54:36.749480 20879 consensus_queue.cc:260] T f07e9b433e554c85b3c25b7e708f8b2f P 94931f4b51c9493cb3d4264ce99f5e76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } }
I20240502 01:54:36.751451 20879 ts_tablet_manager.cc:1398] T f07e9b433e554c85b3c25b7e708f8b2f P 94931f4b51c9493cb3d4264ce99f5e76: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.000s
I20240502 01:54:36.755667 21241 tablet_copy_client.cc:647] T 2c78951695284b1aaab411c0a041c28a P 94931f4b51c9493cb3d4264ce99f5e76: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:36.758466 20783 tablet_bootstrap.cc:492] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 94931f4b51c9493cb3d4264ce99f5e76: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:36.758935 20783 tablet_bootstrap.cc:492] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 94931f4b51c9493cb3d4264ce99f5e76: Bootstrap complete.
I20240502 01:54:36.759672 17764 raft_consensus.cc:1261] T 38a60d7db47149ff90444e5276fc8935 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:36.762897 21241 tablet_copy_client.cc:523] T 2c78951695284b1aaab411c0a041c28a P 94931f4b51c9493cb3d4264ce99f5e76: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:36.764328 20783 ts_tablet_manager.cc:1367] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 94931f4b51c9493cb3d4264ce99f5e76: Time spent bootstrapping tablet: real 0.495s	user 0.049s	sys 0.005s
I20240502 01:54:36.765898 20783 raft_consensus.cc:348] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } }
I20240502 01:54:36.766359 20783 raft_consensus.cc:729] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 94931f4b51c9493cb3d4264ce99f5e76, State: Initialized, Role: LEARNER
I20240502 01:54:36.766798 20783 consensus_queue.cc:260] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 94931f4b51c9493cb3d4264ce99f5e76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } }
I20240502 01:54:36.767774 20783 ts_tablet_manager.cc:1398] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 94931f4b51c9493cb3d4264ce99f5e76: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 01:54:36.769412 17922 tablet_copy_service.cc:342] P da3584f429104d23a1a8899b35cc2138: Request end of tablet copy session 94931f4b51c9493cb3d4264ce99f5e76-60ff2e50b1d74e0c8e0890b87f9ac8ba received from {username='slave'} at 127.9.176.200:36935
I20240502 01:54:36.769568 17922 tablet_copy_service.cc:433] P da3584f429104d23a1a8899b35cc2138: ending tablet copy session 94931f4b51c9493cb3d4264ce99f5e76-60ff2e50b1d74e0c8e0890b87f9ac8ba on tablet 60ff2e50b1d74e0c8e0890b87f9ac8ba with peer 94931f4b51c9493cb3d4264ce99f5e76
I20240502 01:54:36.771167 20790 tablet_bootstrap.cc:492] T 88a137bcfc76410583f910b377d1be7f P 4b7c382c9edd41babdd1a1afc904a813: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:36.771661 20790 tablet_bootstrap.cc:492] T 88a137bcfc76410583f910b377d1be7f P 4b7c382c9edd41babdd1a1afc904a813: Bootstrap complete.
I20240502 01:54:36.771967 20790 ts_tablet_manager.cc:1367] T 88a137bcfc76410583f910b377d1be7f P 4b7c382c9edd41babdd1a1afc904a813: Time spent bootstrapping tablet: real 0.692s	user 0.048s	sys 0.011s
I20240502 01:54:36.772827 20790 raft_consensus.cc:348] T 88a137bcfc76410583f910b377d1be7f P 4b7c382c9edd41babdd1a1afc904a813 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } }
I20240502 01:54:36.773111 20790 raft_consensus.cc:729] T 88a137bcfc76410583f910b377d1be7f P 4b7c382c9edd41babdd1a1afc904a813 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4b7c382c9edd41babdd1a1afc904a813, State: Initialized, Role: LEARNER
I20240502 01:54:36.773310 20790 consensus_queue.cc:260] T 88a137bcfc76410583f910b377d1be7f P 4b7c382c9edd41babdd1a1afc904a813 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } }
I20240502 01:54:36.774699 20790 ts_tablet_manager.cc:1398] T 88a137bcfc76410583f910b377d1be7f P 4b7c382c9edd41babdd1a1afc904a813: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.002s
I20240502 01:54:36.783291 21241 tablet_bootstrap.cc:492] T 2c78951695284b1aaab411c0a041c28a P 94931f4b51c9493cb3d4264ce99f5e76: Bootstrap starting.
I20240502 01:54:36.785405 20911 raft_consensus.cc:2932] T 38a60d7db47149ff90444e5276fc8935 P 1b50a031f2784ba498b3e52e16916205 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } } }
I20240502 01:54:36.789173 19605 raft_consensus.cc:1261] T b40f968655774ce887dff8b7f4651501 P 4b7c382c9edd41babdd1a1afc904a813 [term 1 LEARNER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:36.790258 20628 consensus_queue.cc:1035] T b40f968655774ce887dff8b7f4651501 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:36.792498 19871 raft_consensus.cc:2932] T 38a60d7db47149ff90444e5276fc8935 P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } } }
I20240502 01:54:36.794144 17348 catalog_manager.cc:5484] T 38a60d7db47149ff90444e5276fc8935 P 1b50a031f2784ba498b3e52e16916205 reported cstate change: config changed from index 5 to 6, 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1b50a031f2784ba498b3e52e16916205" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:36.795280 21242 raft_consensus.cc:1013] T b40f968655774ce887dff8b7f4651501 P 1b50a031f2784ba498b3e52e16916205 [term 1 LEADER]: attempt to promote peer 4b7c382c9edd41babdd1a1afc904a813: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 01:54:36.800733 17632 consensus_queue.cc:237] T 38a60d7db47149ff90444e5276fc8935 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } }
I20240502 01:54:36.803591 19738 raft_consensus.cc:1261] T b40f968655774ce887dff8b7f4651501 P 272b2179c1b4476090a89880df2a8baa [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:36.804809 17897 raft_consensus.cc:1261] T b40f968655774ce887dff8b7f4651501 P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:36.805064 21143 raft_consensus.cc:2932] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 84d3a4b174c34e02aedab1e620731359 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 29806d3d37354c169484f6db368f8798 (127.9.176.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } } }
I20240502 01:54:36.805778 20628 consensus_queue.cc:1035] T b40f968655774ce887dff8b7f4651501 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:36.806349 20628 consensus_queue.cc:1035] T b40f968655774ce887dff8b7f4651501 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:36.807965 21186 tablet_bootstrap.cc:492] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 272b2179c1b4476090a89880df2a8baa: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:36.807507 17498 raft_consensus.cc:2932] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 29806d3d37354c169484f6db368f8798 (127.9.176.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } } }
I20240502 01:54:36.808488 21186 tablet_bootstrap.cc:492] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 272b2179c1b4476090a89880df2a8baa: Bootstrap complete.
I20240502 01:54:36.808789 21186 ts_tablet_manager.cc:1367] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 272b2179c1b4476090a89880df2a8baa: Time spent bootstrapping tablet: real 0.364s	user 0.045s	sys 0.015s
I20240502 01:54:36.809757 21186 raft_consensus.cc:348] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 272b2179c1b4476090a89880df2a8baa [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } }
I20240502 01:54:36.810027 21186 raft_consensus.cc:729] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 272b2179c1b4476090a89880df2a8baa [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 272b2179c1b4476090a89880df2a8baa, State: Initialized, Role: LEARNER
I20240502 01:54:36.810284 21186 consensus_queue.cc:260] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 272b2179c1b4476090a89880df2a8baa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } }
I20240502 01:54:36.814747 19871 raft_consensus.cc:1261] T 38a60d7db47149ff90444e5276fc8935 P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:36.816140 20628 consensus_queue.cc:1035] T 38a60d7db47149ff90444e5276fc8935 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:36.816360 21186 ts_tablet_manager.cc:1398] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 272b2179c1b4476090a89880df2a8baa: Time spent starting tablet: real 0.007s	user 0.002s	sys 0.001s
I20240502 01:54:36.817216 17922 tablet_copy_service.cc:342] P da3584f429104d23a1a8899b35cc2138: Request end of tablet copy session 272b2179c1b4476090a89880df2a8baa-60ff2e50b1d74e0c8e0890b87f9ac8ba received from {username='slave'} at 127.9.176.199:46107
I20240502 01:54:36.817363 17922 tablet_copy_service.cc:433] P da3584f429104d23a1a8899b35cc2138: ending tablet copy session 272b2179c1b4476090a89880df2a8baa-60ff2e50b1d74e0c8e0890b87f9ac8ba on tablet 60ff2e50b1d74e0c8e0890b87f9ac8ba with peer 272b2179c1b4476090a89880df2a8baa
I20240502 01:54:36.818126 17348 catalog_manager.cc:5484] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 84d3a4b174c34e02aedab1e620731359 reported cstate change: config changed from index -1 to 4, NON_VOTER 29806d3d37354c169484f6db368f8798 (127.9.176.197) added. New cstate: current_term: 1 leader_uuid: "84d3a4b174c34e02aedab1e620731359" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:36.844540 17764 raft_consensus.cc:1261] T 38a60d7db47149ff90444e5276fc8935 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:36.846010 19467 raft_consensus.cc:2932] T 6da59c0626c64a5988e39a831c42c268 P 29806d3d37354c169484f6db368f8798 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
I20240502 01:54:36.851634 20913 raft_consensus.cc:2932] T b40f968655774ce887dff8b7f4651501 P 1b50a031f2784ba498b3e52e16916205 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 84d3a4b174c34e02aedab1e620731359 (127.9.176.195) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } } }
I20240502 01:54:36.856827 17343 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b40f968655774ce887dff8b7f4651501 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:36.858876 19605 raft_consensus.cc:2932] T b40f968655774ce887dff8b7f4651501 P 4b7c382c9edd41babdd1a1afc904a813 [term 1 LEARNER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 84d3a4b174c34e02aedab1e620731359 (127.9.176.195) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } } }
I20240502 01:54:36.859493 21212 tablet_bootstrap.cc:492] T 37eda67598fe491bbb29aa62b2c4bec7 P 29806d3d37354c169484f6db368f8798: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:36.863716 21212 tablet_bootstrap.cc:492] T 37eda67598fe491bbb29aa62b2c4bec7 P 29806d3d37354c169484f6db368f8798: Bootstrap complete.
I20240502 01:54:36.864456 21212 ts_tablet_manager.cc:1367] T 37eda67598fe491bbb29aa62b2c4bec7 P 29806d3d37354c169484f6db368f8798: Time spent bootstrapping tablet: real 0.295s	user 0.061s	sys 0.004s
I20240502 01:54:36.826532 17497 raft_consensus.cc:2932] T 38a60d7db47149ff90444e5276fc8935 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } } }
I20240502 01:54:36.868068 17350 catalog_manager.cc:5484] T b40f968655774ce887dff8b7f4651501 P 4b7c382c9edd41babdd1a1afc904a813 reported cstate change: config changed from index 6 to 7, VOTER 84d3a4b174c34e02aedab1e620731359 (127.9.176.195) evicted. New cstate: current_term: 1 leader_uuid: "1b50a031f2784ba498b3e52e16916205" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } } }
I20240502 01:54:36.866482 21212 raft_consensus.cc:348] T 37eda67598fe491bbb29aa62b2c4bec7 P 29806d3d37354c169484f6db368f8798 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } }
I20240502 01:54:36.868628 21212 raft_consensus.cc:729] T 37eda67598fe491bbb29aa62b2c4bec7 P 29806d3d37354c169484f6db368f8798 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 29806d3d37354c169484f6db368f8798, State: Initialized, Role: LEARNER
I20240502 01:54:36.868875 21212 consensus_queue.cc:260] T 37eda67598fe491bbb29aa62b2c4bec7 P 29806d3d37354c169484f6db368f8798 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } }
I20240502 01:54:36.869587 21212 ts_tablet_manager.cc:1398] T 37eda67598fe491bbb29aa62b2c4bec7 P 29806d3d37354c169484f6db368f8798: Time spent starting tablet: real 0.004s	user 0.001s	sys 0.000s
I20240502 01:54:36.873801 21242 raft_consensus.cc:1050] T b40f968655774ce887dff8b7f4651501 P 1b50a031f2784ba498b3e52e16916205: attempting to promote NON_VOTER 4b7c382c9edd41babdd1a1afc904a813 to VOTER
I20240502 01:54:36.874043 21176 tablet_bootstrap.cc:492] T 38a60d7db47149ff90444e5276fc8935 P 272b2179c1b4476090a89880df2a8baa: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:36.874549 21242 consensus_queue.cc:237] T b40f968655774ce887dff8b7f4651501 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } }
I20240502 01:54:36.876999 17746 tablet_service.cc:1514] Processing DeleteTablet for tablet b40f968655774ce887dff8b7f4651501 with delete_type TABLET_DATA_TOMBSTONED (TS 84d3a4b174c34e02aedab1e620731359 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:47166
I20240502 01:54:36.877338 21176 tablet_bootstrap.cc:492] T 38a60d7db47149ff90444e5276fc8935 P 272b2179c1b4476090a89880df2a8baa: Bootstrap complete.
I20240502 01:54:36.877694 21176 ts_tablet_manager.cc:1367] T 38a60d7db47149ff90444e5276fc8935 P 272b2179c1b4476090a89880df2a8baa: Time spent bootstrapping tablet: real 0.223s	user 0.051s	sys 0.007s
I20240502 01:54:36.878492 19605 raft_consensus.cc:1261] T b40f968655774ce887dff8b7f4651501 P 4b7c382c9edd41babdd1a1afc904a813 [term 1 LEARNER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:36.878547 21176 raft_consensus.cc:348] T 38a60d7db47149ff90444e5276fc8935 P 272b2179c1b4476090a89880df2a8baa [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } }
I20240502 01:54:36.878859 21176 raft_consensus.cc:729] T 38a60d7db47149ff90444e5276fc8935 P 272b2179c1b4476090a89880df2a8baa [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 272b2179c1b4476090a89880df2a8baa, State: Initialized, Role: LEARNER
I20240502 01:54:36.879144 21176 consensus_queue.cc:260] T 38a60d7db47149ff90444e5276fc8935 P 272b2179c1b4476090a89880df2a8baa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } }
I20240502 01:54:36.879992 21176 ts_tablet_manager.cc:1398] T 38a60d7db47149ff90444e5276fc8935 P 272b2179c1b4476090a89880df2a8baa: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240502 01:54:36.880407 20913 consensus_queue.cc:1035] T b40f968655774ce887dff8b7f4651501 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20240502 01:54:36.876528 17897 raft_consensus.cc:2932] T b40f968655774ce887dff8b7f4651501 P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 84d3a4b174c34e02aedab1e620731359 (127.9.176.195) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } } }
I20240502 01:54:36.881582 17654 tablet_copy_service.cc:342] P 1b50a031f2784ba498b3e52e16916205: Request end of tablet copy session 272b2179c1b4476090a89880df2a8baa-38a60d7db47149ff90444e5276fc8935 received from {username='slave'} at 127.9.176.199:56005
I20240502 01:54:36.881728 17654 tablet_copy_service.cc:433] P 1b50a031f2784ba498b3e52e16916205: ending tablet copy session 272b2179c1b4476090a89880df2a8baa-38a60d7db47149ff90444e5276fc8935 on tablet 38a60d7db47149ff90444e5276fc8935 with peer 272b2179c1b4476090a89880df2a8baa
I20240502 01:54:36.886766 17898 raft_consensus.cc:1261] T b40f968655774ce887dff8b7f4651501 P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:36.887707 20913 consensus_queue.cc:1035] T b40f968655774ce887dff8b7f4651501 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:36.890776 21243 raft_consensus.cc:2932] T b40f968655774ce887dff8b7f4651501 P 1b50a031f2784ba498b3e52e16916205 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
I20240502 01:54:36.894316 21269 tablet_replica.cc:330] T b40f968655774ce887dff8b7f4651501 P 84d3a4b174c34e02aedab1e620731359: stopping tablet replica
I20240502 01:54:36.894547 21269 raft_consensus.cc:2229] T b40f968655774ce887dff8b7f4651501 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:36.894892 21269 raft_consensus.cc:2258] T b40f968655774ce887dff8b7f4651501 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:36.896379 21269 ts_tablet_manager.cc:1875] T b40f968655774ce887dff8b7f4651501 P 84d3a4b174c34e02aedab1e620731359: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:36.897650 19605 raft_consensus.cc:2932] T b40f968655774ce887dff8b7f4651501 P 4b7c382c9edd41babdd1a1afc904a813 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
I20240502 01:54:36.901818 19467 raft_consensus.cc:1261] T 6da59c0626c64a5988e39a831c42c268 P 29806d3d37354c169484f6db368f8798 [term 1 FOLLOWER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:54:36.870565 17520 tablet_copy_service.cc:342] P 311f765f59754bdcb4eadc04e14bd4ea: Request end of tablet copy session 29806d3d37354c169484f6db368f8798-37eda67598fe491bbb29aa62b2c4bec7 received from {username='slave'} at 127.9.176.197:51011
I20240502 01:54:36.904213 17520 tablet_copy_service.cc:433] P 311f765f59754bdcb4eadc04e14bd4ea: ending tablet copy session 29806d3d37354c169484f6db368f8798-37eda67598fe491bbb29aa62b2c4bec7 on tablet 37eda67598fe491bbb29aa62b2c4bec7 with peer 29806d3d37354c169484f6db368f8798
I20240502 01:54:36.913429 17898 raft_consensus.cc:1261] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:36.916275 17898 raft_consensus.cc:2932] T b40f968655774ce887dff8b7f4651501 P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
I20240502 01:54:36.920852 20911 consensus_queue.cc:1035] T 38a60d7db47149ff90444e5276fc8935 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:36.922783 17498 consensus_queue.cc:237] T 49ef898a43704b87ba077cd7b0485e9a P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } }
I20240502 01:54:36.925945 17355 catalog_manager.cc:5484] T b40f968655774ce887dff8b7f4651501 P 4b7c382c9edd41babdd1a1afc904a813 reported cstate change: config changed from index 7 to 8, 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1b50a031f2784ba498b3e52e16916205" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
I20240502 01:54:36.926936 19736 raft_consensus.cc:1261] T b40f968655774ce887dff8b7f4651501 P 272b2179c1b4476090a89880df2a8baa [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:36.927847 21243 consensus_queue.cc:1035] T b40f968655774ce887dff8b7f4651501 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:36.929219 19736 raft_consensus.cc:2932] T b40f968655774ce887dff8b7f4651501 P 272b2179c1b4476090a89880df2a8baa [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 84d3a4b174c34e02aedab1e620731359 (127.9.176.195) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } } }
I20240502 01:54:36.932477 19871 raft_consensus.cc:1203] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
I20240502 01:54:36.934445 17789 tablet_copy_service.cc:342] P 84d3a4b174c34e02aedab1e620731359: Request end of tablet copy session 4b7c382c9edd41babdd1a1afc904a813-88a137bcfc76410583f910b377d1be7f received from {username='slave'} at 127.9.176.198:55177
I20240502 01:54:36.934597 17789 tablet_copy_service.cc:433] P 84d3a4b174c34e02aedab1e620731359: ending tablet copy session 4b7c382c9edd41babdd1a1afc904a813-88a137bcfc76410583f910b377d1be7f on tablet 88a137bcfc76410583f910b377d1be7f with peer 4b7c382c9edd41babdd1a1afc904a813
I20240502 01:54:36.937117 19871 raft_consensus.cc:1203] T f07e9b433e554c85b3c25b7e708f8b2f P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 01:54:36.939144 17789 tablet_copy_service.cc:342] P 84d3a4b174c34e02aedab1e620731359: Request end of tablet copy session 94931f4b51c9493cb3d4264ce99f5e76-f07e9b433e554c85b3c25b7e708f8b2f received from {username='slave'} at 127.9.176.200:34957
I20240502 01:54:36.939293 17789 tablet_copy_service.cc:433] P 84d3a4b174c34e02aedab1e620731359: ending tablet copy session 94931f4b51c9493cb3d4264ce99f5e76-f07e9b433e554c85b3c25b7e708f8b2f on tablet f07e9b433e554c85b3c25b7e708f8b2f with peer 94931f4b51c9493cb3d4264ce99f5e76
I20240502 01:54:36.944911 21143 consensus_queue.cc:1035] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:36.948033 19736 raft_consensus.cc:2932] T b40f968655774ce887dff8b7f4651501 P 272b2179c1b4476090a89880df2a8baa [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
I20240502 01:54:36.950306 21269 ts_tablet_manager.cc:1888] T b40f968655774ce887dff8b7f4651501 P 84d3a4b174c34e02aedab1e620731359: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 01:54:36.950470 21269 log.cc:1196] T b40f968655774ce887dff8b7f4651501 P 84d3a4b174c34e02aedab1e620731359: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/wal/wals/b40f968655774ce887dff8b7f4651501
I20240502 01:54:36.951432 17343 catalog_manager.cc:4830] TS 84d3a4b174c34e02aedab1e620731359 (127.9.176.195:37357): tablet b40f968655774ce887dff8b7f4651501 (table rebalance_test_table_5 [id=73ba62f6c85943bf85dfaaf459d924f0]) successfully deleted
I20240502 01:54:36.955426 17631 consensus_queue.cc:237] T 974f67157aa04d3e8b49b264cfa104e8 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } }
I20240502 01:54:36.962320 19467 raft_consensus.cc:1261] T 974f67157aa04d3e8b49b264cfa104e8 P 29806d3d37354c169484f6db368f8798 [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:36.963586 21243 consensus_queue.cc:1035] T 974f67157aa04d3e8b49b264cfa104e8 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:36.969935 20790 ts_tablet_manager.cc:902] T 17365abbb6c1430aa278719eb8e4a648 P 4b7c382c9edd41babdd1a1afc904a813: Initiating tablet copy from peer da3584f429104d23a1a8899b35cc2138 (127.9.176.196:46381)
I20240502 01:54:36.969676 20913 raft_consensus.cc:2932] T 974f67157aa04d3e8b49b264cfa104e8 P 1b50a031f2784ba498b3e52e16916205 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER da3584f429104d23a1a8899b35cc2138 (127.9.176.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } }
I20240502 01:54:36.976039 17499 raft_consensus.cc:1261] T 38a60d7db47149ff90444e5276fc8935 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:36.976948 21242 consensus_queue.cc:1035] T 38a60d7db47149ff90444e5276fc8935 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:36.978045 17343 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 974f67157aa04d3e8b49b264cfa104e8 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:36.978989 19467 raft_consensus.cc:2932] T 974f67157aa04d3e8b49b264cfa104e8 P 29806d3d37354c169484f6db368f8798 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER da3584f429104d23a1a8899b35cc2138 (127.9.176.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } }
I20240502 01:54:36.981083 21271 ts_tablet_manager.cc:902] T a6b5231a18a5414dbd010d4d48f08dbd P 4b7c382c9edd41babdd1a1afc904a813: Initiating tablet copy from peer 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193:35397)
I20240502 01:54:36.986088 17355 catalog_manager.cc:5484] T 974f67157aa04d3e8b49b264cfa104e8 P 29806d3d37354c169484f6db368f8798 reported cstate change: config changed from index 5 to 6, VOTER da3584f429104d23a1a8899b35cc2138 (127.9.176.196) evicted. New cstate: current_term: 1 leader_uuid: "1b50a031f2784ba498b3e52e16916205" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } }
I20240502 01:54:36.991874 17880 tablet_service.cc:1514] Processing DeleteTablet for tablet 974f67157aa04d3e8b49b264cfa104e8 with delete_type TABLET_DATA_TOMBSTONED (TS da3584f429104d23a1a8899b35cc2138 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:34396
I20240502 01:54:36.992658 21220 tablet_replica.cc:330] T 974f67157aa04d3e8b49b264cfa104e8 P da3584f429104d23a1a8899b35cc2138: stopping tablet replica
I20240502 01:54:36.992949 21220 raft_consensus.cc:2229] T 974f67157aa04d3e8b49b264cfa104e8 P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:36.993122 21271 tablet_copy_client.cc:308] T a6b5231a18a5414dbd010d4d48f08dbd P 4b7c382c9edd41babdd1a1afc904a813: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.193:35397
I20240502 01:54:36.993379 21220 raft_consensus.cc:2258] T 974f67157aa04d3e8b49b264cfa104e8 P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:36.994068 17520 tablet_copy_service.cc:140] P 311f765f59754bdcb4eadc04e14bd4ea: Received BeginTabletCopySession request for tablet a6b5231a18a5414dbd010d4d48f08dbd from peer 4b7c382c9edd41babdd1a1afc904a813 ({username='slave'} at 127.9.176.198:53399)
I20240502 01:54:36.994267 17520 tablet_copy_service.cc:161] P 311f765f59754bdcb4eadc04e14bd4ea: Beginning new tablet copy session on tablet a6b5231a18a5414dbd010d4d48f08dbd from peer 4b7c382c9edd41babdd1a1afc904a813 at {username='slave'} at 127.9.176.198:53399: session id = 4b7c382c9edd41babdd1a1afc904a813-a6b5231a18a5414dbd010d4d48f08dbd
I20240502 01:54:36.995051 21220 ts_tablet_manager.cc:1875] T 974f67157aa04d3e8b49b264cfa104e8 P da3584f429104d23a1a8899b35cc2138: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:36.998499 17520 tablet_copy_source_session.cc:205] T a6b5231a18a5414dbd010d4d48f08dbd P 311f765f59754bdcb4eadc04e14bd4ea: Tablet Copy: opened 0 blocks and 1 log segments
W20240502 01:54:37.003010 17433 consensus_peers.cc:485] T 49ef898a43704b87ba077cd7b0485e9a P 311f765f59754bdcb4eadc04e14bd4ea -> Peer 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200:37831): Couldn't send request to peer 94931f4b51c9493cb3d4264ce99f5e76. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 49ef898a43704b87ba077cd7b0485e9a. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:37.004299 17898 raft_consensus.cc:1261] T 49ef898a43704b87ba077cd7b0485e9a P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Refusing update from remote peer 311f765f59754bdcb4eadc04e14bd4ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:37.004663 17763 raft_consensus.cc:858] T 38a60d7db47149ff90444e5276fc8935 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } }.
  Pending config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } }: 
  New config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } }
W20240502 01:54:37.004802 17763 raft_consensus.cc:1565] T 38a60d7db47149ff90444e5276fc8935 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Could not prepare op 'term: 1 index: 7' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } }.
  Pending config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } }: 
  New config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } }
I20240502 01:54:37.005333 20585 consensus_queue.cc:1035] T 49ef898a43704b87ba077cd7b0485e9a P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:37.006103 17631 raft_consensus.cc:1261] T 49ef898a43704b87ba077cd7b0485e9a P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Refusing update from remote peer 311f765f59754bdcb4eadc04e14bd4ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:37.007885 20585 consensus_queue.cc:1035] T 49ef898a43704b87ba077cd7b0485e9a P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:37.005332 17763 raft_consensus.cc:2932] T 38a60d7db47149ff90444e5276fc8935 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } } }
I20240502 01:54:37.008677 21220 ts_tablet_manager.cc:1888] T 974f67157aa04d3e8b49b264cfa104e8 P da3584f429104d23a1a8899b35cc2138: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:54:37.010509 21220 log.cc:1196] T 974f67157aa04d3e8b49b264cfa104e8 P da3584f429104d23a1a8899b35cc2138: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/wal/wals/974f67157aa04d3e8b49b264cfa104e8
I20240502 01:54:37.011456 17340 catalog_manager.cc:4830] TS da3584f429104d23a1a8899b35cc2138 (127.9.176.196:46381): tablet 974f67157aa04d3e8b49b264cfa104e8 (table rebalance_test_table_9 [id=60aa7e7a81564c3a888cc9b68700c9ad]) successfully deleted
I20240502 01:54:37.016069 17764 raft_consensus.cc:2932] T 974f67157aa04d3e8b49b264cfa104e8 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 29806d3d37354c169484f6db368f8798 (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } }
I20240502 01:54:37.033402 17897 raft_consensus.cc:2932] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 29806d3d37354c169484f6db368f8798 (127.9.176.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } } }
I20240502 01:54:37.034559 20585 raft_consensus.cc:2932] T 49ef898a43704b87ba077cd7b0485e9a P 311f765f59754bdcb4eadc04e14bd4ea [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } } }
I20240502 01:54:37.038621 17633 consensus_queue.cc:237] T 2c78951695284b1aaab411c0a041c28a P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } }
W20240502 01:54:37.042356 17566 consensus_peers.cc:485] T 2c78951695284b1aaab411c0a041c28a P 1b50a031f2784ba498b3e52e16916205 -> Peer 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200:37831): Couldn't send request to peer 94931f4b51c9493cb3d4264ce99f5e76. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:37.043488 17499 raft_consensus.cc:1261] T 2c78951695284b1aaab411c0a041c28a P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:37.043345 17355 catalog_manager.cc:5484] T 49ef898a43704b87ba077cd7b0485e9a P 311f765f59754bdcb4eadc04e14bd4ea reported cstate change: config changed from index -1 to 4, NON_VOTER 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) added. New cstate: current_term: 1 leader_uuid: "311f765f59754bdcb4eadc04e14bd4ea" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:37.042343 17631 raft_consensus.cc:2932] T 49ef898a43704b87ba077cd7b0485e9a P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } } }
I20240502 01:54:37.044981 20628 consensus_queue.cc:1035] T 2c78951695284b1aaab411c0a041c28a P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:37.049238 19736 raft_consensus.cc:1261] T 2c78951695284b1aaab411c0a041c28a P 272b2179c1b4476090a89880df2a8baa [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:37.051154 21242 consensus_queue.cc:1035] T 2c78951695284b1aaab411c0a041c28a P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:37.052204 21243 raft_consensus.cc:2932] T 2c78951695284b1aaab411c0a041c28a P 1b50a031f2784ba498b3e52e16916205 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 84d3a4b174c34e02aedab1e620731359 (127.9.176.195) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } } }
I20240502 01:54:37.053223 17498 raft_consensus.cc:2932] T 2c78951695284b1aaab411c0a041c28a P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 84d3a4b174c34e02aedab1e620731359 (127.9.176.195) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } } }
I20240502 01:54:37.056540 17343 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2c78951695284b1aaab411c0a041c28a with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:37.057569 19738 raft_consensus.cc:2932] T 2c78951695284b1aaab411c0a041c28a P 272b2179c1b4476090a89880df2a8baa [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 84d3a4b174c34e02aedab1e620731359 (127.9.176.195) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } } }
I20240502 01:54:37.059185 20913 raft_consensus.cc:2932] T 38a60d7db47149ff90444e5276fc8935 P 1b50a031f2784ba498b3e52e16916205 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 272b2179c1b4476090a89880df2a8baa (127.9.176.199) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
I20240502 01:54:37.060920 17354 catalog_manager.cc:5484] T 2c78951695284b1aaab411c0a041c28a P 311f765f59754bdcb4eadc04e14bd4ea reported cstate change: config changed from index 6 to 7, VOTER 84d3a4b174c34e02aedab1e620731359 (127.9.176.195) evicted. New cstate: current_term: 1 leader_uuid: "1b50a031f2784ba498b3e52e16916205" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } } }
I20240502 01:54:37.065593 19871 raft_consensus.cc:2932] T 38a60d7db47149ff90444e5276fc8935 P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 272b2179c1b4476090a89880df2a8baa (127.9.176.199) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
I20240502 01:54:37.066850 17498 raft_consensus.cc:2932] T 38a60d7db47149ff90444e5276fc8935 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 272b2179c1b4476090a89880df2a8baa (127.9.176.199) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
I20240502 01:54:37.068107 17343 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 38a60d7db47149ff90444e5276fc8935 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:37.073308 17350 catalog_manager.cc:5484] T 38a60d7db47149ff90444e5276fc8935 P 94931f4b51c9493cb3d4264ce99f5e76 reported cstate change: config changed from index 6 to 7, NON_VOTER 272b2179c1b4476090a89880df2a8baa (127.9.176.199) evicted. New cstate: current_term: 1 leader_uuid: "1b50a031f2784ba498b3e52e16916205" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
I20240502 01:54:37.073761 21271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6b5231a18a5414dbd010d4d48f08dbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:37.074021 17767 raft_consensus.cc:1261] T 974f67157aa04d3e8b49b264cfa104e8 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:37.074542 21143 consensus_queue.cc:1035] T 6da59c0626c64a5988e39a831c42c268 P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240502 01:54:37.075459 17700 consensus_peers.cc:485] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 84d3a4b174c34e02aedab1e620731359 -> Peer 29806d3d37354c169484f6db368f8798 (127.9.176.197:35011): Couldn't send request to peer 29806d3d37354c169484f6db368f8798. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3ca5c4988ff74e3ca8d7a5614bcb93e2. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:37.076895 21243 consensus_queue.cc:1035] T 974f67157aa04d3e8b49b264cfa104e8 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:37.080049 17764 raft_consensus.cc:2932] T 974f67157aa04d3e8b49b264cfa104e8 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER da3584f429104d23a1a8899b35cc2138 (127.9.176.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } }
I20240502 01:54:37.079106 19467 raft_consensus.cc:2932] T 6da59c0626c64a5988e39a831c42c268 P 29806d3d37354c169484f6db368f8798 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER da3584f429104d23a1a8899b35cc2138 (127.9.176.196) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
I20240502 01:54:37.085810 20790 tablet_copy_client.cc:308] T 17365abbb6c1430aa278719eb8e4a648 P 4b7c382c9edd41babdd1a1afc904a813: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.196:46381
I20240502 01:54:37.089103 21271 tablet_copy_client.cc:783] T a6b5231a18a5414dbd010d4d48f08dbd P 4b7c382c9edd41babdd1a1afc904a813: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:37.092113 21271 tablet_copy_client.cc:647] T a6b5231a18a5414dbd010d4d48f08dbd P 4b7c382c9edd41babdd1a1afc904a813: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:37.094043 17922 tablet_copy_service.cc:140] P da3584f429104d23a1a8899b35cc2138: Received BeginTabletCopySession request for tablet 17365abbb6c1430aa278719eb8e4a648 from peer 4b7c382c9edd41babdd1a1afc904a813 ({username='slave'} at 127.9.176.198:45579)
I20240502 01:54:37.094229 17922 tablet_copy_service.cc:161] P da3584f429104d23a1a8899b35cc2138: Beginning new tablet copy session on tablet 17365abbb6c1430aa278719eb8e4a648 from peer 4b7c382c9edd41babdd1a1afc904a813 at {username='slave'} at 127.9.176.198:45579: session id = 4b7c382c9edd41babdd1a1afc904a813-17365abbb6c1430aa278719eb8e4a648
I20240502 01:54:37.096257 17746 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c78951695284b1aaab411c0a041c28a with delete_type TABLET_DATA_TOMBSTONED (TS 84d3a4b174c34e02aedab1e620731359 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:47166
I20240502 01:54:37.096673 21269 tablet_replica.cc:330] T 2c78951695284b1aaab411c0a041c28a P 84d3a4b174c34e02aedab1e620731359: stopping tablet replica
I20240502 01:54:37.096861 21269 raft_consensus.cc:2229] T 2c78951695284b1aaab411c0a041c28a P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:37.097121 21269 raft_consensus.cc:2258] T 2c78951695284b1aaab411c0a041c28a P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:37.098241 21269 ts_tablet_manager.cc:1875] T 2c78951695284b1aaab411c0a041c28a P 84d3a4b174c34e02aedab1e620731359: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:37.108631 21271 tablet_copy_client.cc:523] T a6b5231a18a5414dbd010d4d48f08dbd P 4b7c382c9edd41babdd1a1afc904a813: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:37.108898 17922 tablet_copy_source_session.cc:205] T 17365abbb6c1430aa278719eb8e4a648 P da3584f429104d23a1a8899b35cc2138: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:37.115468 17632 consensus_queue.cc:237] T 38a60d7db47149ff90444e5276fc8935 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } }
I20240502 01:54:37.118294 19871 raft_consensus.cc:1261] T 38a60d7db47149ff90444e5276fc8935 P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:37.118633 17498 raft_consensus.cc:1261] T 38a60d7db47149ff90444e5276fc8935 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:37.118971 21243 consensus_queue.cc:1035] T 38a60d7db47149ff90444e5276fc8935 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:37.119313 20628 consensus_queue.cc:1035] T 38a60d7db47149ff90444e5276fc8935 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:37.121778 21212 ts_tablet_manager.cc:902] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 29806d3d37354c169484f6db368f8798: Initiating tablet copy from peer 84d3a4b174c34e02aedab1e620731359 (127.9.176.195:37357)
I20240502 01:54:37.122141 21243 raft_consensus.cc:2932] T 38a60d7db47149ff90444e5276fc8935 P 1b50a031f2784ba498b3e52e16916205 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 84d3a4b174c34e02aedab1e620731359 (127.9.176.195) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
I20240502 01:54:37.124416 21271 tablet_bootstrap.cc:492] T a6b5231a18a5414dbd010d4d48f08dbd P 4b7c382c9edd41babdd1a1afc904a813: Bootstrap starting.
I20240502 01:54:37.126668 19871 raft_consensus.cc:2932] T 38a60d7db47149ff90444e5276fc8935 P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 84d3a4b174c34e02aedab1e620731359 (127.9.176.195) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
I20240502 01:54:37.129237 21269 ts_tablet_manager.cc:1888] T 2c78951695284b1aaab411c0a041c28a P 84d3a4b174c34e02aedab1e620731359: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 01:54:37.129520 21269 log.cc:1196] T 2c78951695284b1aaab411c0a041c28a P 84d3a4b174c34e02aedab1e620731359: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/wal/wals/2c78951695284b1aaab411c0a041c28a
I20240502 01:54:37.130518 17343 catalog_manager.cc:4830] TS 84d3a4b174c34e02aedab1e620731359 (127.9.176.195:37357): tablet 2c78951695284b1aaab411c0a041c28a (table rebalance_test_table_7 [id=1bd69df0513c4c4db4dd6d6526efee4b]) successfully deleted
I20240502 01:54:37.131042 17343 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 38a60d7db47149ff90444e5276fc8935 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:37.133008 17498 raft_consensus.cc:2932] T 38a60d7db47149ff90444e5276fc8935 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 84d3a4b174c34e02aedab1e620731359 (127.9.176.195) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
I20240502 01:54:37.132943 17355 catalog_manager.cc:5484] T 38a60d7db47149ff90444e5276fc8935 P 1b50a031f2784ba498b3e52e16916205 reported cstate change: config changed from index 7 to 8, VOTER 84d3a4b174c34e02aedab1e620731359 (127.9.176.195) evicted. New cstate: current_term: 1 leader_uuid: "1b50a031f2784ba498b3e52e16916205" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:37.137292 17898 raft_consensus.cc:2932] T 49ef898a43704b87ba077cd7b0485e9a P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } } }
I20240502 01:54:37.137835 17746 tablet_service.cc:1514] Processing DeleteTablet for tablet 38a60d7db47149ff90444e5276fc8935 with delete_type TABLET_DATA_TOMBSTONED (TS 84d3a4b174c34e02aedab1e620731359 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:47166
I20240502 01:54:37.138294 21269 tablet_replica.cc:330] T 38a60d7db47149ff90444e5276fc8935 P 84d3a4b174c34e02aedab1e620731359: stopping tablet replica
I20240502 01:54:37.138541 21269 raft_consensus.cc:2229] T 38a60d7db47149ff90444e5276fc8935 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:37.138952 21269 raft_consensus.cc:2258] T 38a60d7db47149ff90444e5276fc8935 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:37.140211 21269 ts_tablet_manager.cc:1875] T 38a60d7db47149ff90444e5276fc8935 P 84d3a4b174c34e02aedab1e620731359: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:37.146684 21241 tablet_bootstrap.cc:492] T 2c78951695284b1aaab411c0a041c28a P 94931f4b51c9493cb3d4264ce99f5e76: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:37.147241 21241 tablet_bootstrap.cc:492] T 2c78951695284b1aaab411c0a041c28a P 94931f4b51c9493cb3d4264ce99f5e76: Bootstrap complete.
I20240502 01:54:37.148666 19738 raft_consensus.cc:1203] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 272b2179c1b4476090a89880df2a8baa [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
I20240502 01:54:37.148739 20790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17365abbb6c1430aa278719eb8e4a648. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:37.149219 21241 ts_tablet_manager.cc:1367] T 2c78951695284b1aaab411c0a041c28a P 94931f4b51c9493cb3d4264ce99f5e76: Time spent bootstrapping tablet: real 0.366s	user 0.049s	sys 0.015s
I20240502 01:54:37.155041 17898 consensus_queue.cc:237] T a5e03cf44eab41cc842342a138c04067 P da3584f429104d23a1a8899b35cc2138 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } }
I20240502 01:54:37.159300 21241 raft_consensus.cc:348] T 2c78951695284b1aaab411c0a041c28a P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } }
I20240502 01:54:37.159718 21241 raft_consensus.cc:729] T 2c78951695284b1aaab411c0a041c28a P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 94931f4b51c9493cb3d4264ce99f5e76, State: Initialized, Role: LEARNER
I20240502 01:54:37.159987 21241 consensus_queue.cc:260] T 2c78951695284b1aaab411c0a041c28a P 94931f4b51c9493cb3d4264ce99f5e76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } }
I20240502 01:54:37.160701 21241 ts_tablet_manager.cc:1398] T 2c78951695284b1aaab411c0a041c28a P 94931f4b51c9493cb3d4264ce99f5e76: Time spent starting tablet: real 0.011s	user 0.002s	sys 0.000s
I20240502 01:54:37.161547 17654 tablet_copy_service.cc:342] P 1b50a031f2784ba498b3e52e16916205: Request end of tablet copy session 94931f4b51c9493cb3d4264ce99f5e76-2c78951695284b1aaab411c0a041c28a received from {username='slave'} at 127.9.176.200:37843
I20240502 01:54:37.161715 17654 tablet_copy_service.cc:433] P 1b50a031f2784ba498b3e52e16916205: ending tablet copy session 94931f4b51c9493cb3d4264ce99f5e76-2c78951695284b1aaab411c0a041c28a on tablet 2c78951695284b1aaab411c0a041c28a with peer 94931f4b51c9493cb3d4264ce99f5e76
I20240502 01:54:37.182453 19603 raft_consensus.cc:1203] T 88a137bcfc76410583f910b377d1be7f P 4b7c382c9edd41babdd1a1afc904a813 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 01:54:37.213670 17497 raft_consensus.cc:1261] T a5e03cf44eab41cc842342a138c04067 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Refusing update from remote peer da3584f429104d23a1a8899b35cc2138: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:37.214294 21212 tablet_copy_client.cc:308] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 29806d3d37354c169484f6db368f8798: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.195:37357
I20240502 01:54:37.214696 20937 consensus_queue.cc:1035] T a5e03cf44eab41cc842342a138c04067 P da3584f429104d23a1a8899b35cc2138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:37.215559 17789 tablet_copy_service.cc:140] P 84d3a4b174c34e02aedab1e620731359: Received BeginTabletCopySession request for tablet 3ca5c4988ff74e3ca8d7a5614bcb93e2 from peer 29806d3d37354c169484f6db368f8798 ({username='slave'} at 127.9.176.197:36787)
I20240502 01:54:37.215807 17789 tablet_copy_service.cc:161] P 84d3a4b174c34e02aedab1e620731359: Beginning new tablet copy session on tablet 3ca5c4988ff74e3ca8d7a5614bcb93e2 from peer 29806d3d37354c169484f6db368f8798 at {username='slave'} at 127.9.176.197:36787: session id = 29806d3d37354c169484f6db368f8798-3ca5c4988ff74e3ca8d7a5614bcb93e2
I20240502 01:54:37.219269 17789 tablet_copy_source_session.cc:205] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 84d3a4b174c34e02aedab1e620731359: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:37.224781 21212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ca5c4988ff74e3ca8d7a5614bcb93e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:37.225842 17763 raft_consensus.cc:1261] T a5e03cf44eab41cc842342a138c04067 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Refusing update from remote peer da3584f429104d23a1a8899b35cc2138: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:37.227260 20937 consensus_queue.cc:1035] T a5e03cf44eab41cc842342a138c04067 P da3584f429104d23a1a8899b35cc2138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:37.226487 21269 ts_tablet_manager.cc:1888] T 38a60d7db47149ff90444e5276fc8935 P 84d3a4b174c34e02aedab1e620731359: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 01:54:37.232198 21269 log.cc:1196] T 38a60d7db47149ff90444e5276fc8935 P 84d3a4b174c34e02aedab1e620731359: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/wal/wals/38a60d7db47149ff90444e5276fc8935
I20240502 01:54:37.234710 17343 catalog_manager.cc:4830] TS 84d3a4b174c34e02aedab1e620731359 (127.9.176.195:37357): tablet 38a60d7db47149ff90444e5276fc8935 (table rebalance_test_table_5 [id=73ba62f6c85943bf85dfaaf459d924f0]) successfully deleted
I20240502 01:54:37.244565 19718 tablet_service.cc:1514] Processing DeleteTablet for tablet 38a60d7db47149ff90444e5276fc8935 with delete_type TABLET_DATA_TOMBSTONED (TS 272b2179c1b4476090a89880df2a8baa not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:45402
W20240502 01:54:37.259683 17832 consensus_peers.cc:485] T a5e03cf44eab41cc842342a138c04067 P da3584f429104d23a1a8899b35cc2138 -> Peer 29806d3d37354c169484f6db368f8798 (127.9.176.197:35011): Couldn't send request to peer 29806d3d37354c169484f6db368f8798. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a5e03cf44eab41cc842342a138c04067. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:37.261708 20790 tablet_copy_client.cc:783] T 17365abbb6c1430aa278719eb8e4a648 P 4b7c382c9edd41babdd1a1afc904a813: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:37.262109 20790 tablet_copy_client.cc:647] T 17365abbb6c1430aa278719eb8e4a648 P 4b7c382c9edd41babdd1a1afc904a813: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:37.269516 17632 consensus_queue.cc:237] T 0dd8944f9fd542e29072ca501df5e112 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } }
I20240502 01:54:37.277163 19467 raft_consensus.cc:1203] T 37eda67598fe491bbb29aa62b2c4bec7 P 29806d3d37354c169484f6db368f8798 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9]   Dedup: 1.9->[]
I20240502 01:54:37.280535 20790 tablet_copy_client.cc:523] T 17365abbb6c1430aa278719eb8e4a648 P 4b7c382c9edd41babdd1a1afc904a813: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240502 01:54:37.290571 17565 consensus_peers.cc:485] T 0dd8944f9fd542e29072ca501df5e112 P 1b50a031f2784ba498b3e52e16916205 -> Peer 272b2179c1b4476090a89880df2a8baa (127.9.176.199:42269): Couldn't send request to peer 272b2179c1b4476090a89880df2a8baa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0dd8944f9fd542e29072ca501df5e112. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:37.291790 17499 raft_consensus.cc:1261] T 0dd8944f9fd542e29072ca501df5e112 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:37.292181 17763 raft_consensus.cc:1261] T 0dd8944f9fd542e29072ca501df5e112 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:37.292712 20913 consensus_queue.cc:1035] T 0dd8944f9fd542e29072ca501df5e112 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:37.293001 21313 tablet_replica.cc:330] T 38a60d7db47149ff90444e5276fc8935 P 272b2179c1b4476090a89880df2a8baa: stopping tablet replica
I20240502 01:54:37.293515 21313 raft_consensus.cc:2229] T 38a60d7db47149ff90444e5276fc8935 P 272b2179c1b4476090a89880df2a8baa [term 1 LEARNER]: Raft consensus shutting down.
I20240502 01:54:37.293435 20913 consensus_queue.cc:1035] T 0dd8944f9fd542e29072ca501df5e112 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:37.293905 21313 raft_consensus.cc:2258] T 38a60d7db47149ff90444e5276fc8935 P 272b2179c1b4476090a89880df2a8baa [term 1 LEARNER]: Raft consensus is shut down!
I20240502 01:54:37.295251 21313 ts_tablet_manager.cc:1875] T 38a60d7db47149ff90444e5276fc8935 P 272b2179c1b4476090a89880df2a8baa: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:37.298497 20937 raft_consensus.cc:2932] T a5e03cf44eab41cc842342a138c04067 P da3584f429104d23a1a8899b35cc2138 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 29806d3d37354c169484f6db368f8798 (127.9.176.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } } }
I20240502 01:54:37.303970 17497 raft_consensus.cc:2932] T a5e03cf44eab41cc842342a138c04067 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 29806d3d37354c169484f6db368f8798 (127.9.176.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } } }
I20240502 01:54:37.309782 21212 tablet_copy_client.cc:783] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 29806d3d37354c169484f6db368f8798: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:37.310099 21212 tablet_copy_client.cc:647] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 29806d3d37354c169484f6db368f8798: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:37.318814 17355 catalog_manager.cc:5484] T a5e03cf44eab41cc842342a138c04067 P da3584f429104d23a1a8899b35cc2138 reported cstate change: config changed from index -1 to 4, NON_VOTER 29806d3d37354c169484f6db368f8798 (127.9.176.197) added. New cstate: current_term: 1 leader_uuid: "da3584f429104d23a1a8899b35cc2138" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:37.319298 21212 tablet_copy_client.cc:523] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 29806d3d37354c169484f6db368f8798: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:37.335047 21212 tablet_bootstrap.cc:492] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 29806d3d37354c169484f6db368f8798: Bootstrap starting.
I20240502 01:54:37.340811 17767 raft_consensus.cc:2932] T a5e03cf44eab41cc842342a138c04067 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 29806d3d37354c169484f6db368f8798 (127.9.176.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } } }
I20240502 01:54:37.347323 21243 raft_consensus.cc:2932] T 0dd8944f9fd542e29072ca501df5e112 P 1b50a031f2784ba498b3e52e16916205 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 272b2179c1b4476090a89880df2a8baa (127.9.176.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } } }
I20240502 01:54:37.354449 17767 raft_consensus.cc:2932] T 0dd8944f9fd542e29072ca501df5e112 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 272b2179c1b4476090a89880df2a8baa (127.9.176.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } } }
I20240502 01:54:37.361461 17348 catalog_manager.cc:5484] T 0dd8944f9fd542e29072ca501df5e112 P 84d3a4b174c34e02aedab1e620731359 reported cstate change: config changed from index -1 to 4, NON_VOTER 272b2179c1b4476090a89880df2a8baa (127.9.176.199) added. New cstate: current_term: 1 leader_uuid: "1b50a031f2784ba498b3e52e16916205" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } } }
I20240502 01:54:37.371793 17498 raft_consensus.cc:2932] T 0dd8944f9fd542e29072ca501df5e112 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 272b2179c1b4476090a89880df2a8baa (127.9.176.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } } }
I20240502 01:54:37.386430 20790 tablet_bootstrap.cc:492] T 17365abbb6c1430aa278719eb8e4a648 P 4b7c382c9edd41babdd1a1afc904a813: Bootstrap starting.
I20240502 01:54:37.392467 17499 consensus_queue.cc:237] T 62fd543d5c7c41f98926f2148ee61e7d P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } }
I20240502 01:54:37.435647 17763 raft_consensus.cc:1261] T 62fd543d5c7c41f98926f2148ee61e7d P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Refusing update from remote peer 311f765f59754bdcb4eadc04e14bd4ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:37.436398 20585 consensus_queue.cc:1035] T 62fd543d5c7c41f98926f2148ee61e7d P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:37.445412 17632 raft_consensus.cc:1261] T 62fd543d5c7c41f98926f2148ee61e7d P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Refusing update from remote peer 311f765f59754bdcb4eadc04e14bd4ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:37.460872 20585 consensus_queue.cc:1035] T 62fd543d5c7c41f98926f2148ee61e7d P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:37.461627 20585 raft_consensus.cc:2932] T 62fd543d5c7c41f98926f2148ee61e7d P 311f765f59754bdcb4eadc04e14bd4ea [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 272b2179c1b4476090a89880df2a8baa (127.9.176.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } } }
I20240502 01:54:37.465852 21271 tablet_bootstrap.cc:492] T a6b5231a18a5414dbd010d4d48f08dbd P 4b7c382c9edd41babdd1a1afc904a813: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:37.466516 21271 tablet_bootstrap.cc:492] T a6b5231a18a5414dbd010d4d48f08dbd P 4b7c382c9edd41babdd1a1afc904a813: Bootstrap complete.
I20240502 01:54:37.466945 21271 ts_tablet_manager.cc:1367] T a6b5231a18a5414dbd010d4d48f08dbd P 4b7c382c9edd41babdd1a1afc904a813: Time spent bootstrapping tablet: real 0.343s	user 0.087s	sys 0.050s
I20240502 01:54:37.468950 21053 raft_consensus.cc:1050] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P da3584f429104d23a1a8899b35cc2138: attempting to promote NON_VOTER 94931f4b51c9493cb3d4264ce99f5e76 to VOTER
I20240502 01:54:37.470042 21053 consensus_queue.cc:237] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P da3584f429104d23a1a8899b35cc2138 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } }
I20240502 01:54:37.470120 21271 raft_consensus.cc:348] T a6b5231a18a5414dbd010d4d48f08dbd P 4b7c382c9edd41babdd1a1afc904a813 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } }
I20240502 01:54:37.470412 21271 raft_consensus.cc:729] T a6b5231a18a5414dbd010d4d48f08dbd P 4b7c382c9edd41babdd1a1afc904a813 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4b7c382c9edd41babdd1a1afc904a813, State: Initialized, Role: LEARNER
I20240502 01:54:37.470692 21271 consensus_queue.cc:260] T a6b5231a18a5414dbd010d4d48f08dbd P 4b7c382c9edd41babdd1a1afc904a813 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } }
I20240502 01:54:37.471366 21271 ts_tablet_manager.cc:1398] T a6b5231a18a5414dbd010d4d48f08dbd P 4b7c382c9edd41babdd1a1afc904a813: Time spent starting tablet: real 0.004s	user 0.001s	sys 0.001s
I20240502 01:54:37.474311 17520 tablet_copy_service.cc:342] P 311f765f59754bdcb4eadc04e14bd4ea: Request end of tablet copy session 4b7c382c9edd41babdd1a1afc904a813-a6b5231a18a5414dbd010d4d48f08dbd received from {username='slave'} at 127.9.176.198:53399
I20240502 01:54:37.474467 17520 tablet_copy_service.cc:433] P 311f765f59754bdcb4eadc04e14bd4ea: ending tablet copy session 4b7c382c9edd41babdd1a1afc904a813-a6b5231a18a5414dbd010d4d48f08dbd on tablet a6b5231a18a5414dbd010d4d48f08dbd with peer 4b7c382c9edd41babdd1a1afc904a813
I20240502 01:54:37.487342 17355 catalog_manager.cc:5484] T 62fd543d5c7c41f98926f2148ee61e7d P 311f765f59754bdcb4eadc04e14bd4ea reported cstate change: config changed from index -1 to 4, NON_VOTER 272b2179c1b4476090a89880df2a8baa (127.9.176.199) added. New cstate: current_term: 1 leader_uuid: "311f765f59754bdcb4eadc04e14bd4ea" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:37.507588 21241 ts_tablet_manager.cc:902] T 49ef898a43704b87ba077cd7b0485e9a P 94931f4b51c9493cb3d4264ce99f5e76: Initiating tablet copy from peer 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193:35397)
I20240502 01:54:37.510552 21140 raft_consensus.cc:1050] T f07e9b433e554c85b3c25b7e708f8b2f P 84d3a4b174c34e02aedab1e620731359: attempting to promote NON_VOTER 94931f4b51c9493cb3d4264ce99f5e76 to VOTER
I20240502 01:54:37.511631 21140 consensus_queue.cc:237] T f07e9b433e554c85b3c25b7e708f8b2f P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } }
I20240502 01:54:37.518790 21212 tablet_bootstrap.cc:492] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 29806d3d37354c169484f6db368f8798: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:37.519321 21212 tablet_bootstrap.cc:492] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 29806d3d37354c169484f6db368f8798: Bootstrap complete.
I20240502 01:54:37.519671 21212 ts_tablet_manager.cc:1367] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 29806d3d37354c169484f6db368f8798: Time spent bootstrapping tablet: real 0.185s	user 0.052s	sys 0.006s
I20240502 01:54:37.520746 21212 raft_consensus.cc:348] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 29806d3d37354c169484f6db368f8798 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } }
I20240502 01:54:37.521065 21212 raft_consensus.cc:729] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 29806d3d37354c169484f6db368f8798 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 29806d3d37354c169484f6db368f8798, State: Initialized, Role: LEARNER
I20240502 01:54:37.521354 21212 consensus_queue.cc:260] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 29806d3d37354c169484f6db368f8798 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } }
I20240502 01:54:37.521908 21212 ts_tablet_manager.cc:1398] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 29806d3d37354c169484f6db368f8798: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 01:54:37.522908 17789 tablet_copy_service.cc:342] P 84d3a4b174c34e02aedab1e620731359: Request end of tablet copy session 29806d3d37354c169484f6db368f8798-3ca5c4988ff74e3ca8d7a5614bcb93e2 received from {username='slave'} at 127.9.176.197:36787
I20240502 01:54:37.523090 17789 tablet_copy_service.cc:433] P 84d3a4b174c34e02aedab1e620731359: ending tablet copy session 29806d3d37354c169484f6db368f8798-3ca5c4988ff74e3ca8d7a5614bcb93e2 on tablet 3ca5c4988ff74e3ca8d7a5614bcb93e2 with peer 29806d3d37354c169484f6db368f8798
I20240502 01:54:37.538642 19872 raft_consensus.cc:1261] T f07e9b433e554c85b3c25b7e708f8b2f P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 LEARNER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:37.539448 21007 consensus_queue.cc:1035] T f07e9b433e554c85b3c25b7e708f8b2f P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:37.543151 19603 raft_consensus.cc:1203] T a6b5231a18a5414dbd010d4d48f08dbd P 4b7c382c9edd41babdd1a1afc904a813 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 01:54:37.557574 17631 consensus_queue.cc:237] T b40f968655774ce887dff8b7f4651501 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } }
W20240502 01:54:37.559087 17432 consensus_peers.cc:485] T 62fd543d5c7c41f98926f2148ee61e7d P 311f765f59754bdcb4eadc04e14bd4ea -> Peer 272b2179c1b4476090a89880df2a8baa (127.9.176.199:42269): Couldn't send request to peer 272b2179c1b4476090a89880df2a8baa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 62fd543d5c7c41f98926f2148ee61e7d. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:37.560554 19738 raft_consensus.cc:1261] T b40f968655774ce887dff8b7f4651501 P 272b2179c1b4476090a89880df2a8baa [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:54:37.561214 19603 raft_consensus.cc:1261] T b40f968655774ce887dff8b7f4651501 P 4b7c382c9edd41babdd1a1afc904a813 [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:54:37.561319 21243 consensus_queue.cc:1035] T b40f968655774ce887dff8b7f4651501 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:37.562505 21243 consensus_queue.cc:1035] T b40f968655774ce887dff8b7f4651501 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:37.564638 17763 raft_consensus.cc:2932] T 62fd543d5c7c41f98926f2148ee61e7d P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 272b2179c1b4476090a89880df2a8baa (127.9.176.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } } }
I20240502 01:54:37.565586 21243 raft_consensus.cc:2932] T b40f968655774ce887dff8b7f4651501 P 1b50a031f2784ba498b3e52e16916205 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER da3584f429104d23a1a8899b35cc2138 (127.9.176.196) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
I20240502 01:54:37.566294 21313 ts_tablet_manager.cc:1888] T 38a60d7db47149ff90444e5276fc8935 P 272b2179c1b4476090a89880df2a8baa: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:54:37.566516 21313 log.cc:1196] T 38a60d7db47149ff90444e5276fc8935 P 272b2179c1b4476090a89880df2a8baa: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-6/wal/wals/38a60d7db47149ff90444e5276fc8935
I20240502 01:54:37.568671 19738 raft_consensus.cc:2932] T b40f968655774ce887dff8b7f4651501 P 272b2179c1b4476090a89880df2a8baa [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER da3584f429104d23a1a8899b35cc2138 (127.9.176.196) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
I20240502 01:54:37.574113 17340 catalog_manager.cc:4830] TS 272b2179c1b4476090a89880df2a8baa (127.9.176.199:42269): tablet 38a60d7db47149ff90444e5276fc8935 (table rebalance_test_table_5 [id=73ba62f6c85943bf85dfaaf459d924f0]) successfully deleted
I20240502 01:54:37.582296 17350 catalog_manager.cc:5484] T b40f968655774ce887dff8b7f4651501 P 1b50a031f2784ba498b3e52e16916205 reported cstate change: config changed from index 8 to 9, VOTER da3584f429104d23a1a8899b35cc2138 (127.9.176.196) evicted. New cstate: current_term: 1 leader_uuid: "1b50a031f2784ba498b3e52e16916205" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:37.586521 19603 raft_consensus.cc:2932] T b40f968655774ce887dff8b7f4651501 P 4b7c382c9edd41babdd1a1afc904a813 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER da3584f429104d23a1a8899b35cc2138 (127.9.176.196) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
I20240502 01:54:37.589289 17499 raft_consensus.cc:1261] T f07e9b433e554c85b3c25b7e708f8b2f P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:37.589287 17631 raft_consensus.cc:1261] T f07e9b433e554c85b3c25b7e708f8b2f P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:37.590251 21007 consensus_queue.cc:1035] T f07e9b433e554c85b3c25b7e708f8b2f P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:37.590921 21007 consensus_queue.cc:1035] T f07e9b433e554c85b3c25b7e708f8b2f P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:37.597584 17880 tablet_service.cc:1514] Processing DeleteTablet for tablet b40f968655774ce887dff8b7f4651501 with delete_type TABLET_DATA_TOMBSTONED (TS da3584f429104d23a1a8899b35cc2138 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:34396
I20240502 01:54:37.604032 21357 tablet_replica.cc:330] T b40f968655774ce887dff8b7f4651501 P da3584f429104d23a1a8899b35cc2138: stopping tablet replica
I20240502 01:54:37.604548 21357 raft_consensus.cc:2229] T b40f968655774ce887dff8b7f4651501 P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:37.606583 21140 raft_consensus.cc:2932] T f07e9b433e554c85b3c25b7e708f8b2f P 84d3a4b174c34e02aedab1e620731359 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
I20240502 01:54:37.613570 17499 raft_consensus.cc:2932] T f07e9b433e554c85b3c25b7e708f8b2f P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
I20240502 01:54:37.614405 17633 consensus_queue.cc:237] T 0dd8944f9fd542e29072ca501df5e112 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } }
W20240502 01:54:37.620070 17565 consensus_peers.cc:485] T 0dd8944f9fd542e29072ca501df5e112 P 1b50a031f2784ba498b3e52e16916205 -> Peer 272b2179c1b4476090a89880df2a8baa (127.9.176.199:42269): Couldn't send request to peer 272b2179c1b4476090a89880df2a8baa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0dd8944f9fd542e29072ca501df5e112. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:37.620708 17763 raft_consensus.cc:1261] T 0dd8944f9fd542e29072ca501df5e112 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240502 01:54:37.621078 17565 consensus_peers.cc:485] T 0dd8944f9fd542e29072ca501df5e112 P 1b50a031f2784ba498b3e52e16916205 -> Peer 29806d3d37354c169484f6db368f8798 (127.9.176.197:35011): Couldn't send request to peer 29806d3d37354c169484f6db368f8798. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0dd8944f9fd542e29072ca501df5e112. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:37.621366 21243 consensus_queue.cc:1035] T 0dd8944f9fd542e29072ca501df5e112 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:37.622737 17498 raft_consensus.cc:1261] T 0dd8944f9fd542e29072ca501df5e112 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:37.624315 21243 consensus_queue.cc:1035] T 0dd8944f9fd542e29072ca501df5e112 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:37.625046 19872 raft_consensus.cc:2932] T f07e9b433e554c85b3c25b7e708f8b2f P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
I20240502 01:54:37.626495 20628 raft_consensus.cc:2932] T 0dd8944f9fd542e29072ca501df5e112 P 1b50a031f2784ba498b3e52e16916205 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 29806d3d37354c169484f6db368f8798 (127.9.176.197) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } } }
I20240502 01:54:37.627223 17767 raft_consensus.cc:2932] T 0dd8944f9fd542e29072ca501df5e112 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 29806d3d37354c169484f6db368f8798 (127.9.176.197) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } } }
I20240502 01:54:37.628311 21007 raft_consensus.cc:1050] T 88a137bcfc76410583f910b377d1be7f P 84d3a4b174c34e02aedab1e620731359: attempting to promote NON_VOTER 4b7c382c9edd41babdd1a1afc904a813 to VOTER
I20240502 01:54:37.629535 21007 consensus_queue.cc:237] T 88a137bcfc76410583f910b377d1be7f P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } }
I20240502 01:54:37.633888 17500 raft_consensus.cc:2932] T 0dd8944f9fd542e29072ca501df5e112 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 29806d3d37354c169484f6db368f8798 (127.9.176.197) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } } }
I20240502 01:54:37.638875 17355 catalog_manager.cc:5484] T f07e9b433e554c85b3c25b7e708f8b2f P 311f765f59754bdcb4eadc04e14bd4ea reported cstate change: config changed from index 4 to 5, 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "84d3a4b174c34e02aedab1e620731359" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
I20240502 01:54:37.644328 19603 raft_consensus.cc:1261] T 88a137bcfc76410583f910b377d1be7f P 4b7c382c9edd41babdd1a1afc904a813 [term 1 LEARNER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:37.644546 17351 catalog_manager.cc:5484] T 0dd8944f9fd542e29072ca501df5e112 P 1b50a031f2784ba498b3e52e16916205 reported cstate change: config changed from index 4 to 5, NON_VOTER 29806d3d37354c169484f6db368f8798 (127.9.176.197) added. New cstate: current_term: 1 leader_uuid: "1b50a031f2784ba498b3e52e16916205" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:37.644685 17631 raft_consensus.cc:1261] T 88a137bcfc76410583f910b377d1be7f P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:37.646031 21363 consensus_queue.cc:1035] T 88a137bcfc76410583f910b377d1be7f P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:37.646878 21007 consensus_queue.cc:1035] T 88a137bcfc76410583f910b377d1be7f P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:37.648792 17500 raft_consensus.cc:1261] T 88a137bcfc76410583f910b377d1be7f P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:37.648841 19871 raft_consensus.cc:2932] T 2c78951695284b1aaab411c0a041c28a P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 LEARNER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 84d3a4b174c34e02aedab1e620731359 (127.9.176.195) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } } }
I20240502 01:54:37.653759 21007 consensus_queue.cc:1035] T 88a137bcfc76410583f910b377d1be7f P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:37.655443 21357 raft_consensus.cc:2258] T b40f968655774ce887dff8b7f4651501 P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:37.656023 19870 raft_consensus.cc:1261] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 LEARNER]: Refusing update from remote peer da3584f429104d23a1a8899b35cc2138: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:37.656886 20937 consensus_queue.cc:1035] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P da3584f429104d23a1a8899b35cc2138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:37.657191 21357 ts_tablet_manager.cc:1875] T b40f968655774ce887dff8b7f4651501 P da3584f429104d23a1a8899b35cc2138: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:37.663033 17767 raft_consensus.cc:1261] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Refusing update from remote peer da3584f429104d23a1a8899b35cc2138: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:37.664893 20915 consensus_queue.cc:1035] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P da3584f429104d23a1a8899b35cc2138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:37.668524 21357 ts_tablet_manager.cc:1888] T b40f968655774ce887dff8b7f4651501 P da3584f429104d23a1a8899b35cc2138: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 01:54:37.670111 21357 log.cc:1196] T b40f968655774ce887dff8b7f4651501 P da3584f429104d23a1a8899b35cc2138: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/wal/wals/b40f968655774ce887dff8b7f4651501
I20240502 01:54:37.670213 20915 raft_consensus.cc:2932] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P da3584f429104d23a1a8899b35cc2138 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } } }
I20240502 01:54:37.672461 17340 catalog_manager.cc:4830] TS da3584f429104d23a1a8899b35cc2138 (127.9.176.196:46381): tablet b40f968655774ce887dff8b7f4651501 (table rebalance_test_table_5 [id=73ba62f6c85943bf85dfaaf459d924f0]) successfully deleted
I20240502 01:54:37.673203 17633 raft_consensus.cc:2932] T f07e9b433e554c85b3c25b7e708f8b2f P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
I20240502 01:54:37.681210 17767 raft_consensus.cc:2932] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } } }
I20240502 01:54:37.684890 19738 raft_consensus.cc:1261] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 272b2179c1b4476090a89880df2a8baa [term 1 LEARNER]: Refusing update from remote peer da3584f429104d23a1a8899b35cc2138: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:37.685503 21373 consensus_queue.cc:1035] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P da3584f429104d23a1a8899b35cc2138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:37.687700 17499 raft_consensus.cc:1261] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Refusing update from remote peer da3584f429104d23a1a8899b35cc2138: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:37.688822 20915 consensus_queue.cc:1035] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P da3584f429104d23a1a8899b35cc2138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:37.690560 17343 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b40f968655774ce887dff8b7f4651501 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:37.669157 17632 raft_consensus.cc:2932] T 62fd543d5c7c41f98926f2148ee61e7d P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 272b2179c1b4476090a89880df2a8baa (127.9.176.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } } }
I20240502 01:54:37.700663 17347 catalog_manager.cc:5484] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P da3584f429104d23a1a8899b35cc2138 reported cstate change: config changed from index 5 to 6, 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "da3584f429104d23a1a8899b35cc2138" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:37.703873 19738 raft_consensus.cc:2932] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 272b2179c1b4476090a89880df2a8baa [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } } }
I20240502 01:54:37.704972 17499 raft_consensus.cc:2932] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } } }
I20240502 01:54:37.711510 21373 raft_consensus.cc:1050] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P da3584f429104d23a1a8899b35cc2138: attempting to promote NON_VOTER 272b2179c1b4476090a89880df2a8baa to VOTER
I20240502 01:54:37.712678 21373 consensus_queue.cc:237] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P da3584f429104d23a1a8899b35cc2138 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } }
I20240502 01:54:37.716102 17767 raft_consensus.cc:1261] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Refusing update from remote peer da3584f429104d23a1a8899b35cc2138: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:37.716213 19870 raft_consensus.cc:1261] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Refusing update from remote peer da3584f429104d23a1a8899b35cc2138: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:37.716728 19738 raft_consensus.cc:1261] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 272b2179c1b4476090a89880df2a8baa [term 1 LEARNER]: Refusing update from remote peer da3584f429104d23a1a8899b35cc2138: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:37.717268 21373 consensus_queue.cc:1035] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P da3584f429104d23a1a8899b35cc2138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:37.717573 20915 consensus_queue.cc:1035] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P da3584f429104d23a1a8899b35cc2138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:37.717929 17499 raft_consensus.cc:1261] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Refusing update from remote peer da3584f429104d23a1a8899b35cc2138: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:37.720446 19467 raft_consensus.cc:1203] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 29806d3d37354c169484f6db368f8798 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 01:54:37.720511 20915 consensus_queue.cc:1035] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P da3584f429104d23a1a8899b35cc2138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:37.721946 19870 raft_consensus.cc:2932] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } } }
I20240502 01:54:37.726276 21375 raft_consensus.cc:2932] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P da3584f429104d23a1a8899b35cc2138 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 272b2179c1b4476090a89880df2a8baa (127.9.176.199) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } }
I20240502 01:54:37.730235 21371 consensus_queue.cc:1035] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P da3584f429104d23a1a8899b35cc2138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:37.732247 17499 raft_consensus.cc:2932] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 272b2179c1b4476090a89880df2a8baa (127.9.176.199) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } }
I20240502 01:54:37.738595 19870 raft_consensus.cc:2932] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 272b2179c1b4476090a89880df2a8baa (127.9.176.199) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } }
I20240502 01:54:37.744185 17767 raft_consensus.cc:2932] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 272b2179c1b4476090a89880df2a8baa (127.9.176.199) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } }
I20240502 01:54:37.744819 19738 raft_consensus.cc:2932] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 272b2179c1b4476090a89880df2a8baa [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 272b2179c1b4476090a89880df2a8baa (127.9.176.199) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } }
I20240502 01:54:37.754837 21354 raft_consensus.cc:2932] T 88a137bcfc76410583f910b377d1be7f P 84d3a4b174c34e02aedab1e620731359 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
W20240502 01:54:37.755812 17340 catalog_manager.cc:5007] ChangeConfig:REMOVE_PEER RPC for tablet 60ff2e50b1d74e0c8e0890b87f9ac8ba with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER failed with leader da3584f429104d23a1a8899b35cc2138 (127.9.176.196:46381) due to CAS failure; no further retry: Illegal state: Request specified cas_config_opid_index of 6 but the committed config has opid_index of 7
I20240502 01:54:37.759486 17347 catalog_manager.cc:5484] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 94931f4b51c9493cb3d4264ce99f5e76 reported cstate change: config changed from index 6 to 7, 272b2179c1b4476090a89880df2a8baa (127.9.176.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "da3584f429104d23a1a8899b35cc2138" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } }
I20240502 01:54:37.763499 17631 raft_consensus.cc:2932] T 88a137bcfc76410583f910b377d1be7f P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
I20240502 01:54:37.763988 17498 raft_consensus.cc:2932] T 88a137bcfc76410583f910b377d1be7f P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
I20240502 01:54:37.772498 17898 consensus_queue.cc:237] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P da3584f429104d23a1a8899b35cc2138 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } }
I20240502 01:54:37.772841 17347 catalog_manager.cc:5484] T 88a137bcfc76410583f910b377d1be7f P 84d3a4b174c34e02aedab1e620731359 reported cstate change: config changed from index 4 to 5, 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "84d3a4b174c34e02aedab1e620731359" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:37.776196 19738 raft_consensus.cc:1261] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 272b2179c1b4476090a89880df2a8baa [term 1 FOLLOWER]: Refusing update from remote peer da3584f429104d23a1a8899b35cc2138: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:37.776453 17767 raft_consensus.cc:1261] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Refusing update from remote peer da3584f429104d23a1a8899b35cc2138: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:37.777303 21373 consensus_queue.cc:1035] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P da3584f429104d23a1a8899b35cc2138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:37.777844 20915 consensus_queue.cc:1035] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P da3584f429104d23a1a8899b35cc2138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:37.778568 19871 raft_consensus.cc:1261] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Refusing update from remote peer da3584f429104d23a1a8899b35cc2138: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:37.779345 20915 consensus_queue.cc:1035] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P da3584f429104d23a1a8899b35cc2138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:37.782886 21053 raft_consensus.cc:2932] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P da3584f429104d23a1a8899b35cc2138 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } }
I20240502 01:54:37.786702 21241 tablet_copy_client.cc:308] T 49ef898a43704b87ba077cd7b0485e9a P 94931f4b51c9493cb3d4264ce99f5e76: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.193:35397
I20240502 01:54:37.788201 17520 tablet_copy_service.cc:140] P 311f765f59754bdcb4eadc04e14bd4ea: Received BeginTabletCopySession request for tablet 49ef898a43704b87ba077cd7b0485e9a from peer 94931f4b51c9493cb3d4264ce99f5e76 ({username='slave'} at 127.9.176.200:46111)
I20240502 01:54:37.788404 17520 tablet_copy_service.cc:161] P 311f765f59754bdcb4eadc04e14bd4ea: Beginning new tablet copy session on tablet 49ef898a43704b87ba077cd7b0485e9a from peer 94931f4b51c9493cb3d4264ce99f5e76 at {username='slave'} at 127.9.176.200:46111: session id = 94931f4b51c9493cb3d4264ce99f5e76-49ef898a43704b87ba077cd7b0485e9a
I20240502 01:54:37.791163 17340 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 60ff2e50b1d74e0c8e0890b87f9ac8ba with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:37.792106 17520 tablet_copy_source_session.cc:205] T 49ef898a43704b87ba077cd7b0485e9a P 311f765f59754bdcb4eadc04e14bd4ea: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:37.792143 17767 raft_consensus.cc:2932] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } }
I20240502 01:54:37.794003 21241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49ef898a43704b87ba077cd7b0485e9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:37.803973 19738 raft_consensus.cc:2932] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 272b2179c1b4476090a89880df2a8baa [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } }
I20240502 01:54:37.806648 17767 consensus_queue.cc:237] T f07e9b433e554c85b3c25b7e708f8b2f P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } }
I20240502 01:54:37.806744 17499 consensus_queue.cc:237] T 075ded9523e44c3d92844805338643a3 P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } }
I20240502 01:54:37.807723 21212 ts_tablet_manager.cc:902] T a5e03cf44eab41cc842342a138c04067 P 29806d3d37354c169484f6db368f8798: Initiating tablet copy from peer da3584f429104d23a1a8899b35cc2138 (127.9.176.196:46381)
I20240502 01:54:37.816385 17631 raft_consensus.cc:1261] T f07e9b433e554c85b3c25b7e708f8b2f P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:37.816624 19871 raft_consensus.cc:1261] T f07e9b433e554c85b3c25b7e708f8b2f P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:37.816517 19870 raft_consensus.cc:2932] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } }
I20240502 01:54:37.817271 21354 consensus_queue.cc:1035] T f07e9b433e554c85b3c25b7e708f8b2f P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:37.817651 21361 consensus_queue.cc:1035] T f07e9b433e554c85b3c25b7e708f8b2f P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:37.805817 17354 catalog_manager.cc:5484] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 84d3a4b174c34e02aedab1e620731359 reported cstate change: config changed from index 7 to 8, VOTER 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193) evicted. New cstate: current_term: 1 leader_uuid: "da3584f429104d23a1a8899b35cc2138" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } }
I20240502 01:54:37.820667 21143 raft_consensus.cc:2932] T f07e9b433e554c85b3c25b7e708f8b2f P 84d3a4b174c34e02aedab1e620731359 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
I20240502 01:54:37.825425 21241 tablet_copy_client.cc:783] T 49ef898a43704b87ba077cd7b0485e9a P 94931f4b51c9493cb3d4264ce99f5e76: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:37.825712 21241 tablet_copy_client.cc:647] T 49ef898a43704b87ba077cd7b0485e9a P 94931f4b51c9493cb3d4264ce99f5e76: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:37.827512 17343 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f07e9b433e554c85b3c25b7e708f8b2f with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:37.827121 17631 raft_consensus.cc:2932] T f07e9b433e554c85b3c25b7e708f8b2f P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
W20240502 01:54:37.828153 17433 consensus_peers.cc:485] T 075ded9523e44c3d92844805338643a3 P 311f765f59754bdcb4eadc04e14bd4ea -> Peer 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200:37831): Couldn't send request to peer 94931f4b51c9493cb3d4264ce99f5e76. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 075ded9523e44c3d92844805338643a3. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:37.828572 21241 tablet_copy_client.cc:523] T 49ef898a43704b87ba077cd7b0485e9a P 94931f4b51c9493cb3d4264ce99f5e76: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:37.831800 17477 tablet_service.cc:1514] Processing DeleteTablet for tablet 60ff2e50b1d74e0c8e0890b87f9ac8ba with delete_type TABLET_DATA_TOMBSTONED (TS 311f765f59754bdcb4eadc04e14bd4ea not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:48974
I20240502 01:54:37.831971 17898 raft_consensus.cc:1261] T 075ded9523e44c3d92844805338643a3 P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Refusing update from remote peer 311f765f59754bdcb4eadc04e14bd4ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:37.832732 20788 consensus_queue.cc:1035] T 075ded9523e44c3d92844805338643a3 P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:37.837848 17632 raft_consensus.cc:1261] T 075ded9523e44c3d92844805338643a3 P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Refusing update from remote peer 311f765f59754bdcb4eadc04e14bd4ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:37.838555 20788 consensus_queue.cc:1035] T 075ded9523e44c3d92844805338643a3 P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:37.848613 20788 raft_consensus.cc:2932] T 075ded9523e44c3d92844805338643a3 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } } }
I20240502 01:54:37.849442 19870 raft_consensus.cc:2932] T f07e9b433e554c85b3c25b7e708f8b2f P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
I20240502 01:54:37.849951 17352 catalog_manager.cc:5484] T f07e9b433e554c85b3c25b7e708f8b2f P 1b50a031f2784ba498b3e52e16916205 reported cstate change: config changed from index 5 to 6, VOTER 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193) evicted. New cstate: current_term: 1 leader_uuid: "84d3a4b174c34e02aedab1e620731359" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
I20240502 01:54:37.853852 17631 raft_consensus.cc:2932] T 075ded9523e44c3d92844805338643a3 P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } } }
I20240502 01:54:37.856201 17355 catalog_manager.cc:5484] T 075ded9523e44c3d92844805338643a3 P 311f765f59754bdcb4eadc04e14bd4ea reported cstate change: config changed from index -1 to 4, NON_VOTER 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) added. New cstate: current_term: 1 leader_uuid: "311f765f59754bdcb4eadc04e14bd4ea" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:37.870823 19605 raft_consensus.cc:2932] T 88a137bcfc76410583f910b377d1be7f P 4b7c382c9edd41babdd1a1afc904a813 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
I20240502 01:54:37.872812 17898 raft_consensus.cc:2932] T 075ded9523e44c3d92844805338643a3 P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } } }
I20240502 01:54:37.873286 21402 tablet_replica.cc:330] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 311f765f59754bdcb4eadc04e14bd4ea: stopping tablet replica
I20240502 01:54:37.873548 21402 raft_consensus.cc:2229] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:37.874047 21402 raft_consensus.cc:2258] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:37.874153 17478 tablet_service.cc:1514] Processing DeleteTablet for tablet f07e9b433e554c85b3c25b7e708f8b2f with delete_type TABLET_DATA_TOMBSTONED (TS 311f765f59754bdcb4eadc04e14bd4ea not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:48974
I20240502 01:54:37.884790 21342 raft_consensus.cc:1050] T 37eda67598fe491bbb29aa62b2c4bec7 P 311f765f59754bdcb4eadc04e14bd4ea: attempting to promote NON_VOTER 29806d3d37354c169484f6db368f8798 to VOTER
I20240502 01:54:37.886071 21342 consensus_queue.cc:237] T 37eda67598fe491bbb29aa62b2c4bec7 P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } }
I20240502 01:54:37.890033 21402 ts_tablet_manager.cc:1875] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 311f765f59754bdcb4eadc04e14bd4ea: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:37.896706 20790 tablet_bootstrap.cc:492] T 17365abbb6c1430aa278719eb8e4a648 P 4b7c382c9edd41babdd1a1afc904a813: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:37.896956 19467 raft_consensus.cc:1261] T 37eda67598fe491bbb29aa62b2c4bec7 P 29806d3d37354c169484f6db368f8798 [term 1 LEARNER]: Refusing update from remote peer 311f765f59754bdcb4eadc04e14bd4ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:54:37.898361 20585 consensus_queue.cc:1035] T 37eda67598fe491bbb29aa62b2c4bec7 P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:54:37.900868 17898 consensus_queue.cc:237] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P da3584f429104d23a1a8899b35cc2138 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } }
I20240502 01:54:37.901741 19870 raft_consensus.cc:1261] T 37eda67598fe491bbb29aa62b2c4bec7 P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Refusing update from remote peer 311f765f59754bdcb4eadc04e14bd4ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:54:37.902482 20788 consensus_queue.cc:1035] T 37eda67598fe491bbb29aa62b2c4bec7 P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:54:37.907054 19738 raft_consensus.cc:1261] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 272b2179c1b4476090a89880df2a8baa [term 1 FOLLOWER]: Refusing update from remote peer da3584f429104d23a1a8899b35cc2138: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:54:37.907899 20915 consensus_queue.cc:1035] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P da3584f429104d23a1a8899b35cc2138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:37.909256 21243 raft_consensus.cc:1050] T 2c78951695284b1aaab411c0a041c28a P 1b50a031f2784ba498b3e52e16916205: attempting to promote NON_VOTER 94931f4b51c9493cb3d4264ce99f5e76 to VOTER
I20240502 01:54:37.910161 21243 consensus_queue.cc:237] T 2c78951695284b1aaab411c0a041c28a P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } }
I20240502 01:54:37.910939 20915 raft_consensus.cc:2932] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P da3584f429104d23a1a8899b35cc2138 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 84d3a4b174c34e02aedab1e620731359 (127.9.176.195) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } }
I20240502 01:54:37.912338 19738 raft_consensus.cc:2932] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 272b2179c1b4476090a89880df2a8baa [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 84d3a4b174c34e02aedab1e620731359 (127.9.176.195) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } }
I20240502 01:54:37.913198 19736 raft_consensus.cc:1261] T 2c78951695284b1aaab411c0a041c28a P 272b2179c1b4476090a89880df2a8baa [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:37.913856 17499 raft_consensus.cc:1261] T 2c78951695284b1aaab411c0a041c28a P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:37.913825 21243 consensus_queue.cc:1035] T 2c78951695284b1aaab411c0a041c28a P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:37.914206 19871 raft_consensus.cc:1261] T 2c78951695284b1aaab411c0a041c28a P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 LEARNER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:37.914698 20913 consensus_queue.cc:1035] T 2c78951695284b1aaab411c0a041c28a P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:37.915477 20628 consensus_queue.cc:1035] T 2c78951695284b1aaab411c0a041c28a P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:37.919826 21212 tablet_copy_client.cc:308] T a5e03cf44eab41cc842342a138c04067 P 29806d3d37354c169484f6db368f8798: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.196:46381
I20240502 01:54:37.921340 20628 raft_consensus.cc:2932] T 2c78951695284b1aaab411c0a041c28a P 1b50a031f2784ba498b3e52e16916205 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
I20240502 01:54:37.924818 17499 raft_consensus.cc:2932] T 2c78951695284b1aaab411c0a041c28a P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
I20240502 01:54:37.930171 19871 raft_consensus.cc:2932] T 2c78951695284b1aaab411c0a041c28a P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
I20240502 01:54:37.937669 21241 tablet_bootstrap.cc:492] T 49ef898a43704b87ba077cd7b0485e9a P 94931f4b51c9493cb3d4264ce99f5e76: Bootstrap starting.
I20240502 01:54:37.938040 17767 consensus_queue.cc:237] T 88a137bcfc76410583f910b377d1be7f P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } }
I20240502 01:54:37.941470 19736 raft_consensus.cc:2932] T 2c78951695284b1aaab411c0a041c28a P 272b2179c1b4476090a89880df2a8baa [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
I20240502 01:54:37.944415 17631 raft_consensus.cc:1261] T 37eda67598fe491bbb29aa62b2c4bec7 P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Refusing update from remote peer 311f765f59754bdcb4eadc04e14bd4ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:54:37.945117 21342 consensus_queue.cc:1035] T 37eda67598fe491bbb29aa62b2c4bec7 P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:54:37.947374 17354 catalog_manager.cc:5484] T 2c78951695284b1aaab411c0a041c28a P 1b50a031f2784ba498b3e52e16916205 reported cstate change: config changed from index 7 to 8, 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1b50a031f2784ba498b3e52e16916205" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:37.950901 17499 raft_consensus.cc:1261] T 88a137bcfc76410583f910b377d1be7f P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:37.951714 21361 consensus_queue.cc:1035] T 88a137bcfc76410583f910b377d1be7f P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:37.955299 21361 raft_consensus.cc:2932] T 88a137bcfc76410583f910b377d1be7f P 84d3a4b174c34e02aedab1e620731359 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1b50a031f2784ba498b3e52e16916205 (127.9.176.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
I20240502 01:54:37.957597 17348 catalog_manager.cc:5484] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 272b2179c1b4476090a89880df2a8baa reported cstate change: config changed from index 8 to 9, VOTER 84d3a4b174c34e02aedab1e620731359 (127.9.176.195) evicted. New cstate: current_term: 1 leader_uuid: "da3584f429104d23a1a8899b35cc2138" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } }
I20240502 01:54:37.959381 17500 raft_consensus.cc:2932] T 88a137bcfc76410583f910b377d1be7f P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1b50a031f2784ba498b3e52e16916205 (127.9.176.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
I20240502 01:54:37.965317 17746 tablet_service.cc:1514] Processing DeleteTablet for tablet 60ff2e50b1d74e0c8e0890b87f9ac8ba with delete_type TABLET_DATA_TOMBSTONED (TS 84d3a4b174c34e02aedab1e620731359 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:47166
I20240502 01:54:37.973670 17348 catalog_manager.cc:5484] T 88a137bcfc76410583f910b377d1be7f P 84d3a4b174c34e02aedab1e620731359 reported cstate change: config changed from index 5 to 6, VOTER 1b50a031f2784ba498b3e52e16916205 (127.9.176.194) evicted. New cstate: current_term: 1 leader_uuid: "84d3a4b174c34e02aedab1e620731359" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:37.975605 17343 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 88a137bcfc76410583f910b377d1be7f with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:37.976974 21422 tablet_replica.cc:330] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 84d3a4b174c34e02aedab1e620731359: stopping tablet replica
I20240502 01:54:37.977381 21422 raft_consensus.cc:2229] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:37.978076 21422 raft_consensus.cc:2258] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:37.988699 21422 ts_tablet_manager.cc:1875] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 84d3a4b174c34e02aedab1e620731359: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:37.992146 20788 raft_consensus.cc:2932] T 37eda67598fe491bbb29aa62b2c4bec7 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 29806d3d37354c169484f6db368f8798 (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } }
I20240502 01:54:37.994138 17613 tablet_service.cc:1514] Processing DeleteTablet for tablet 88a137bcfc76410583f910b377d1be7f with delete_type TABLET_DATA_TOMBSTONED (TS 1b50a031f2784ba498b3e52e16916205 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41484
I20240502 01:54:37.998134 21424 ts_tablet_manager.cc:902] T 62fd543d5c7c41f98926f2148ee61e7d P 272b2179c1b4476090a89880df2a8baa: Initiating tablet copy from peer 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193:35397)
I20240502 01:54:37.998080 19467 raft_consensus.cc:2932] T 37eda67598fe491bbb29aa62b2c4bec7 P 29806d3d37354c169484f6db368f8798 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 29806d3d37354c169484f6db368f8798 (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } }
I20240502 01:54:38.006095 17631 raft_consensus.cc:2932] T 37eda67598fe491bbb29aa62b2c4bec7 P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 29806d3d37354c169484f6db368f8798 (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } }
I20240502 01:54:38.006928 17355 catalog_manager.cc:5484] T 37eda67598fe491bbb29aa62b2c4bec7 P 311f765f59754bdcb4eadc04e14bd4ea reported cstate change: config changed from index 9 to 10, 29806d3d37354c169484f6db368f8798 (127.9.176.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "311f765f59754bdcb4eadc04e14bd4ea" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:38.009505 19871 raft_consensus.cc:1261] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Refusing update from remote peer da3584f429104d23a1a8899b35cc2138: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:54:38.012634 20915 consensus_queue.cc:1035] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P da3584f429104d23a1a8899b35cc2138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:38.014492 17340 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 60ff2e50b1d74e0c8e0890b87f9ac8ba with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:38.015113 19871 raft_consensus.cc:2932] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 84d3a4b174c34e02aedab1e620731359 (127.9.176.195) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } }
I20240502 01:54:38.024601 21427 tablet_replica.cc:330] T 88a137bcfc76410583f910b377d1be7f P 1b50a031f2784ba498b3e52e16916205: stopping tablet replica
I20240502 01:54:37.947805 19603 raft_consensus.cc:1261] T 88a137bcfc76410583f910b377d1be7f P 4b7c382c9edd41babdd1a1afc904a813 [term 1 FOLLOWER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:38.026364 20790 tablet_bootstrap.cc:492] T 17365abbb6c1430aa278719eb8e4a648 P 4b7c382c9edd41babdd1a1afc904a813: Bootstrap complete.
I20240502 01:54:38.026355 21362 consensus_queue.cc:1035] T 88a137bcfc76410583f910b377d1be7f P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:38.026748 20790 ts_tablet_manager.cc:1367] T 17365abbb6c1430aa278719eb8e4a648 P 4b7c382c9edd41babdd1a1afc904a813: Time spent bootstrapping tablet: real 0.640s	user 0.052s	sys 0.005s
I20240502 01:54:38.028002 20790 raft_consensus.cc:348] T 17365abbb6c1430aa278719eb8e4a648 P 4b7c382c9edd41babdd1a1afc904a813 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } }
I20240502 01:54:38.028285 20790 raft_consensus.cc:729] T 17365abbb6c1430aa278719eb8e4a648 P 4b7c382c9edd41babdd1a1afc904a813 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4b7c382c9edd41babdd1a1afc904a813, State: Initialized, Role: LEARNER
I20240502 01:54:38.028548 20790 consensus_queue.cc:260] T 17365abbb6c1430aa278719eb8e4a648 P 4b7c382c9edd41babdd1a1afc904a813 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } }
I20240502 01:54:38.028425 19603 raft_consensus.cc:2932] T 88a137bcfc76410583f910b377d1be7f P 4b7c382c9edd41babdd1a1afc904a813 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1b50a031f2784ba498b3e52e16916205 (127.9.176.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
I20240502 01:54:38.029189 20790 ts_tablet_manager.cc:1398] T 17365abbb6c1430aa278719eb8e4a648 P 4b7c382c9edd41babdd1a1afc904a813: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240502 01:54:38.034794 17922 tablet_copy_service.cc:342] P da3584f429104d23a1a8899b35cc2138: Request end of tablet copy session 4b7c382c9edd41babdd1a1afc904a813-17365abbb6c1430aa278719eb8e4a648 received from {username='slave'} at 127.9.176.198:45579
I20240502 01:54:38.034977 17922 tablet_copy_service.cc:433] P da3584f429104d23a1a8899b35cc2138: ending tablet copy session 4b7c382c9edd41babdd1a1afc904a813-17365abbb6c1430aa278719eb8e4a648 on tablet 17365abbb6c1430aa278719eb8e4a648 with peer 4b7c382c9edd41babdd1a1afc904a813
I20240502 01:54:38.038964 21422 ts_tablet_manager.cc:1888] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 84d3a4b174c34e02aedab1e620731359: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 01:54:38.039047 21427 raft_consensus.cc:2229] T 88a137bcfc76410583f910b377d1be7f P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:38.039364 21427 raft_consensus.cc:2258] T 88a137bcfc76410583f910b377d1be7f P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:38.041083 21433 ts_tablet_manager.cc:902] T 0dd8944f9fd542e29072ca501df5e112 P 272b2179c1b4476090a89880df2a8baa: Initiating tablet copy from peer 1b50a031f2784ba498b3e52e16916205 (127.9.176.194:45617)
I20240502 01:54:38.045042 21427 ts_tablet_manager.cc:1875] T 88a137bcfc76410583f910b377d1be7f P 1b50a031f2784ba498b3e52e16916205: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:38.046113 21424 tablet_copy_client.cc:308] T 62fd543d5c7c41f98926f2148ee61e7d P 272b2179c1b4476090a89880df2a8baa: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.193:35397
I20240502 01:54:38.046562 21433 tablet_copy_client.cc:308] T 0dd8944f9fd542e29072ca501df5e112 P 272b2179c1b4476090a89880df2a8baa: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.194:45617
I20240502 01:54:38.047672 17654 tablet_copy_service.cc:140] P 1b50a031f2784ba498b3e52e16916205: Received BeginTabletCopySession request for tablet 0dd8944f9fd542e29072ca501df5e112 from peer 272b2179c1b4476090a89880df2a8baa ({username='slave'} at 127.9.176.199:56005)
I20240502 01:54:38.047881 17654 tablet_copy_service.cc:161] P 1b50a031f2784ba498b3e52e16916205: Beginning new tablet copy session on tablet 0dd8944f9fd542e29072ca501df5e112 from peer 272b2179c1b4476090a89880df2a8baa at {username='slave'} at 127.9.176.199:56005: session id = 272b2179c1b4476090a89880df2a8baa-0dd8944f9fd542e29072ca501df5e112
I20240502 01:54:38.052690 17654 tablet_copy_source_session.cc:205] T 0dd8944f9fd542e29072ca501df5e112 P 1b50a031f2784ba498b3e52e16916205: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:38.059986 17922 tablet_copy_service.cc:140] P da3584f429104d23a1a8899b35cc2138: Received BeginTabletCopySession request for tablet a5e03cf44eab41cc842342a138c04067 from peer 29806d3d37354c169484f6db368f8798 ({username='slave'} at 127.9.176.197:44989)
I20240502 01:54:38.060289 17922 tablet_copy_service.cc:161] P da3584f429104d23a1a8899b35cc2138: Beginning new tablet copy session on tablet a5e03cf44eab41cc842342a138c04067 from peer 29806d3d37354c169484f6db368f8798 at {username='slave'} at 127.9.176.197:44989: session id = 29806d3d37354c169484f6db368f8798-a5e03cf44eab41cc842342a138c04067
I20240502 01:54:38.065641 17922 tablet_copy_source_session.cc:205] T a5e03cf44eab41cc842342a138c04067 P da3584f429104d23a1a8899b35cc2138: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:38.067672 21212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5e03cf44eab41cc842342a138c04067. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:38.074726 21422 log.cc:1196] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 84d3a4b174c34e02aedab1e620731359: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/wal/wals/60ff2e50b1d74e0c8e0890b87f9ac8ba
I20240502 01:54:38.076181 21212 tablet_copy_client.cc:783] T a5e03cf44eab41cc842342a138c04067 P 29806d3d37354c169484f6db368f8798: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:38.076763 21212 tablet_copy_client.cc:647] T a5e03cf44eab41cc842342a138c04067 P 29806d3d37354c169484f6db368f8798: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:38.077551 17343 catalog_manager.cc:4830] TS 84d3a4b174c34e02aedab1e620731359 (127.9.176.195:37357): tablet 60ff2e50b1d74e0c8e0890b87f9ac8ba (table rebalance_test_table_6 [id=d49bf19a92cf45ceadf02d01dbdae6ad]) successfully deleted
I20240502 01:54:38.079998 19872 raft_consensus.cc:2932] T 37eda67598fe491bbb29aa62b2c4bec7 P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 29806d3d37354c169484f6db368f8798 (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } }
I20240502 01:54:38.087100 17500 consensus_queue.cc:237] T 37eda67598fe491bbb29aa62b2c4bec7 P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } }
I20240502 01:54:38.087873 21212 tablet_copy_client.cc:523] T a5e03cf44eab41cc842342a138c04067 P 29806d3d37354c169484f6db368f8798: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:38.090395 19467 raft_consensus.cc:1261] T 37eda67598fe491bbb29aa62b2c4bec7 P 29806d3d37354c169484f6db368f8798 [term 1 FOLLOWER]: Refusing update from remote peer 311f765f59754bdcb4eadc04e14bd4ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 01:54:38.091153 21342 consensus_queue.cc:1035] T 37eda67598fe491bbb29aa62b2c4bec7 P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 01:54:38.094305 21342 raft_consensus.cc:2932] T 37eda67598fe491bbb29aa62b2c4bec7 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 1b50a031f2784ba498b3e52e16916205 (127.9.176.194) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } }
I20240502 01:54:38.095237 21402 ts_tablet_manager.cc:1888] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 311f765f59754bdcb4eadc04e14bd4ea: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240502 01:54:38.095402 21402 log.cc:1196] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 311f765f59754bdcb4eadc04e14bd4ea: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/wal/wals/60ff2e50b1d74e0c8e0890b87f9ac8ba
I20240502 01:54:38.096403 21402 tablet_replica.cc:330] T f07e9b433e554c85b3c25b7e708f8b2f P 311f765f59754bdcb4eadc04e14bd4ea: stopping tablet replica
I20240502 01:54:38.097435 19872 raft_consensus.cc:1261] T 37eda67598fe491bbb29aa62b2c4bec7 P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Refusing update from remote peer 311f765f59754bdcb4eadc04e14bd4ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 01:54:38.097754 17343 catalog_manager.cc:4830] TS 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193:35397): tablet 60ff2e50b1d74e0c8e0890b87f9ac8ba (table rebalance_test_table_6 [id=d49bf19a92cf45ceadf02d01dbdae6ad]) successfully deleted
I20240502 01:54:38.098774 20788 consensus_queue.cc:1035] T 37eda67598fe491bbb29aa62b2c4bec7 P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 01:54:38.100883 17343 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 37eda67598fe491bbb29aa62b2c4bec7 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:38.101365 19872 raft_consensus.cc:2932] T 37eda67598fe491bbb29aa62b2c4bec7 P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 1b50a031f2784ba498b3e52e16916205 (127.9.176.194) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } }
I20240502 01:54:38.103848 17354 catalog_manager.cc:5484] T 37eda67598fe491bbb29aa62b2c4bec7 P 311f765f59754bdcb4eadc04e14bd4ea reported cstate change: config changed from index 10 to 11, VOTER 1b50a031f2784ba498b3e52e16916205 (127.9.176.194) evicted. New cstate: current_term: 1 leader_uuid: "311f765f59754bdcb4eadc04e14bd4ea" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:38.108897 17613 tablet_service.cc:1514] Processing DeleteTablet for tablet 37eda67598fe491bbb29aa62b2c4bec7 with delete_type TABLET_DATA_TOMBSTONED (TS 1b50a031f2784ba498b3e52e16916205 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:41484
I20240502 01:54:38.120791 17631 consensus_queue.cc:237] T 91e0e82526c0489a8908e250bf3c851f P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } }
I20240502 01:54:38.143548 21433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dd8944f9fd542e29072ca501df5e112. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 01:54:38.147220 17566 consensus_peers.cc:485] T 91e0e82526c0489a8908e250bf3c851f P 1b50a031f2784ba498b3e52e16916205 -> Peer 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200:37831): Couldn't send request to peer 94931f4b51c9493cb3d4264ce99f5e76. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 91e0e82526c0489a8908e250bf3c851f. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:38.149586 21342 raft_consensus.cc:1050] T a6b5231a18a5414dbd010d4d48f08dbd P 311f765f59754bdcb4eadc04e14bd4ea: attempting to promote NON_VOTER 4b7c382c9edd41babdd1a1afc904a813 to VOTER
I20240502 01:54:38.149749 17500 raft_consensus.cc:1261] T 91e0e82526c0489a8908e250bf3c851f P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:38.150633 20628 consensus_queue.cc:1035] T 91e0e82526c0489a8908e250bf3c851f P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:38.150806 21342 consensus_queue.cc:237] T a6b5231a18a5414dbd010d4d48f08dbd P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } }
I20240502 01:54:38.151057 21433 tablet_copy_client.cc:783] T 0dd8944f9fd542e29072ca501df5e112 P 272b2179c1b4476090a89880df2a8baa: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:38.151396 21433 tablet_copy_client.cc:647] T 0dd8944f9fd542e29072ca501df5e112 P 272b2179c1b4476090a89880df2a8baa: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:38.154819 21433 tablet_copy_client.cc:523] T 0dd8944f9fd542e29072ca501df5e112 P 272b2179c1b4476090a89880df2a8baa: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:38.148638 19465 raft_consensus.cc:1261] T 91e0e82526c0489a8908e250bf3c851f P 29806d3d37354c169484f6db368f8798 [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:38.101500 19467 raft_consensus.cc:2932] T 37eda67598fe491bbb29aa62b2c4bec7 P 29806d3d37354c169484f6db368f8798 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 1b50a031f2784ba498b3e52e16916205 (127.9.176.194) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } }
I20240502 01:54:38.160888 20628 consensus_queue.cc:1035] T 91e0e82526c0489a8908e250bf3c851f P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:38.098567 21212 tablet_bootstrap.cc:492] T a5e03cf44eab41cc842342a138c04067 P 29806d3d37354c169484f6db368f8798: Bootstrap starting.
I20240502 01:54:38.168911 21402 raft_consensus.cc:2229] T f07e9b433e554c85b3c25b7e708f8b2f P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:38.169389 21402 raft_consensus.cc:2258] T f07e9b433e554c85b3c25b7e708f8b2f P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:38.170629 21402 ts_tablet_manager.cc:1875] T f07e9b433e554c85b3c25b7e708f8b2f P 311f765f59754bdcb4eadc04e14bd4ea: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:38.173398 17499 consensus_queue.cc:237] T 7ce9d4281c1b46258493feb39e8d9ab2 P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } }
I20240502 01:54:38.176208 21433 tablet_bootstrap.cc:492] T 0dd8944f9fd542e29072ca501df5e112 P 272b2179c1b4476090a89880df2a8baa: Bootstrap starting.
I20240502 01:54:38.176275 20628 raft_consensus.cc:2932] T 91e0e82526c0489a8908e250bf3c851f P 1b50a031f2784ba498b3e52e16916205 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } } }
I20240502 01:54:38.182103 19467 raft_consensus.cc:2932] T 91e0e82526c0489a8908e250bf3c851f P 29806d3d37354c169484f6db368f8798 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } } }
I20240502 01:54:38.183842 17520 tablet_copy_service.cc:140] P 311f765f59754bdcb4eadc04e14bd4ea: Received BeginTabletCopySession request for tablet 62fd543d5c7c41f98926f2148ee61e7d from peer 272b2179c1b4476090a89880df2a8baa ({username='slave'} at 127.9.176.199:51681)
I20240502 01:54:38.184082 17520 tablet_copy_service.cc:161] P 311f765f59754bdcb4eadc04e14bd4ea: Beginning new tablet copy session on tablet 62fd543d5c7c41f98926f2148ee61e7d from peer 272b2179c1b4476090a89880df2a8baa at {username='slave'} at 127.9.176.199:51681: session id = 272b2179c1b4476090a89880df2a8baa-62fd543d5c7c41f98926f2148ee61e7d
I20240502 01:54:38.195241 17520 tablet_copy_source_session.cc:205] T 62fd543d5c7c41f98926f2148ee61e7d P 311f765f59754bdcb4eadc04e14bd4ea: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:38.198130 21424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62fd543d5c7c41f98926f2148ee61e7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:38.201007 17354 catalog_manager.cc:5484] T 91e0e82526c0489a8908e250bf3c851f P 1b50a031f2784ba498b3e52e16916205 reported cstate change: config changed from index 6 to 7, NON_VOTER 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) added. New cstate: current_term: 1 leader_uuid: "1b50a031f2784ba498b3e52e16916205" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:38.202661 21402 ts_tablet_manager.cc:1888] T f07e9b433e554c85b3c25b7e708f8b2f P 311f765f59754bdcb4eadc04e14bd4ea: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:54:38.204618 21402 log.cc:1196] T f07e9b433e554c85b3c25b7e708f8b2f P 311f765f59754bdcb4eadc04e14bd4ea: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/wal/wals/f07e9b433e554c85b3c25b7e708f8b2f
I20240502 01:54:38.205308 21424 tablet_copy_client.cc:783] T 62fd543d5c7c41f98926f2148ee61e7d P 272b2179c1b4476090a89880df2a8baa: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:38.205577 21424 tablet_copy_client.cc:647] T 62fd543d5c7c41f98926f2148ee61e7d P 272b2179c1b4476090a89880df2a8baa: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:38.209311 17898 raft_consensus.cc:1261] T 7ce9d4281c1b46258493feb39e8d9ab2 P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Refusing update from remote peer 311f765f59754bdcb4eadc04e14bd4ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 01:54:38.210316 17432 consensus_peers.cc:485] T 7ce9d4281c1b46258493feb39e8d9ab2 P 311f765f59754bdcb4eadc04e14bd4ea -> Peer 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198:36865): Couldn't send request to peer 4b7c382c9edd41babdd1a1afc904a813. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7ce9d4281c1b46258493feb39e8d9ab2. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:38.211138 17898 raft_consensus.cc:1261] T a6b5231a18a5414dbd010d4d48f08dbd P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Refusing update from remote peer 311f765f59754bdcb4eadc04e14bd4ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:38.211405 17631 raft_consensus.cc:1261] T a6b5231a18a5414dbd010d4d48f08dbd P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Refusing update from remote peer 311f765f59754bdcb4eadc04e14bd4ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:38.211767 20788 consensus_queue.cc:1035] T 7ce9d4281c1b46258493feb39e8d9ab2 P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:38.211817 17343 catalog_manager.cc:4830] TS 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193:35397): tablet f07e9b433e554c85b3c25b7e708f8b2f (table rebalance_test_table_4 [id=14b25b3f4ede4b65870815b014d90449]) successfully deleted
I20240502 01:54:38.207737 21424 tablet_copy_client.cc:523] T 62fd543d5c7c41f98926f2148ee61e7d P 272b2179c1b4476090a89880df2a8baa: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:38.213323 17633 raft_consensus.cc:1261] T 7ce9d4281c1b46258493feb39e8d9ab2 P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Refusing update from remote peer 311f765f59754bdcb4eadc04e14bd4ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:38.213397 19603 raft_consensus.cc:1261] T a6b5231a18a5414dbd010d4d48f08dbd P 4b7c382c9edd41babdd1a1afc904a813 [term 1 LEARNER]: Refusing update from remote peer 311f765f59754bdcb4eadc04e14bd4ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:38.214093 20585 consensus_queue.cc:1035] T 7ce9d4281c1b46258493feb39e8d9ab2 P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:38.216753 21424 tablet_bootstrap.cc:492] T 62fd543d5c7c41f98926f2148ee61e7d P 272b2179c1b4476090a89880df2a8baa: Bootstrap starting.
I20240502 01:54:38.225561 21427 ts_tablet_manager.cc:1888] T 88a137bcfc76410583f910b377d1be7f P 1b50a031f2784ba498b3e52e16916205: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:54:38.235496 20585 raft_consensus.cc:2932] T 7ce9d4281c1b46258493feb39e8d9ab2 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } } }
I20240502 01:54:38.239362 21427 log.cc:1196] T 88a137bcfc76410583f910b377d1be7f P 1b50a031f2784ba498b3e52e16916205: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/wal/wals/88a137bcfc76410583f910b377d1be7f
I20240502 01:54:38.240813 17901 raft_consensus.cc:2932] T 7ce9d4281c1b46258493feb39e8d9ab2 P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } } }
I20240502 01:54:38.246443 17347 catalog_manager.cc:5484] T 7ce9d4281c1b46258493feb39e8d9ab2 P 311f765f59754bdcb4eadc04e14bd4ea reported cstate change: config changed from index -1 to 4, NON_VOTER 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) added. New cstate: current_term: 1 leader_uuid: "311f765f59754bdcb4eadc04e14bd4ea" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:38.248123 17633 raft_consensus.cc:2932] T 7ce9d4281c1b46258493feb39e8d9ab2 P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } } }
I20240502 01:54:38.252085 17343 catalog_manager.cc:4830] TS 1b50a031f2784ba498b3e52e16916205 (127.9.176.194:45617): tablet 88a137bcfc76410583f910b377d1be7f (table rebalance_test_table_3 [id=c5e1575e337a4256a90ce5b685b3ff28]) successfully deleted
I20240502 01:54:38.259934 21427 tablet_replica.cc:330] T 37eda67598fe491bbb29aa62b2c4bec7 P 1b50a031f2784ba498b3e52e16916205: stopping tablet replica
I20240502 01:54:38.263018 17499 raft_consensus.cc:2932] T 91e0e82526c0489a8908e250bf3c851f P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } } }
I20240502 01:54:38.265563 21427 raft_consensus.cc:2229] T 37eda67598fe491bbb29aa62b2c4bec7 P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:38.267882 21427 raft_consensus.cc:2258] T 37eda67598fe491bbb29aa62b2c4bec7 P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:38.274119 20788 consensus_queue.cc:1035] T a6b5231a18a5414dbd010d4d48f08dbd P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:38.275254 21342 consensus_queue.cc:1035] T a6b5231a18a5414dbd010d4d48f08dbd P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:38.276425 21409 consensus_queue.cc:1035] T a6b5231a18a5414dbd010d4d48f08dbd P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:38.291074 21427 ts_tablet_manager.cc:1875] T 37eda67598fe491bbb29aa62b2c4bec7 P 1b50a031f2784ba498b3e52e16916205: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:38.308867 21462 ts_tablet_manager.cc:902] T 0dd8944f9fd542e29072ca501df5e112 P 29806d3d37354c169484f6db368f8798: Initiating tablet copy from peer 1b50a031f2784ba498b3e52e16916205 (127.9.176.194:45617)
I20240502 01:54:38.312520 21342 raft_consensus.cc:2932] T a6b5231a18a5414dbd010d4d48f08dbd P 311f765f59754bdcb4eadc04e14bd4ea [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
I20240502 01:54:38.318459 21427 ts_tablet_manager.cc:1888] T 37eda67598fe491bbb29aa62b2c4bec7 P 1b50a031f2784ba498b3e52e16916205: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.10
I20240502 01:54:38.318703 21362 raft_consensus.cc:1050] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 84d3a4b174c34e02aedab1e620731359: attempting to promote NON_VOTER 29806d3d37354c169484f6db368f8798 to VOTER
I20240502 01:54:38.319774 21362 consensus_queue.cc:237] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } }
I20240502 01:54:38.323776 19603 raft_consensus.cc:2932] T a6b5231a18a5414dbd010d4d48f08dbd P 4b7c382c9edd41babdd1a1afc904a813 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
I20240502 01:54:38.324263 19605 raft_consensus.cc:1203] T 17365abbb6c1430aa278719eb8e4a648 P 4b7c382c9edd41babdd1a1afc904a813 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 01:54:38.325882 17354 catalog_manager.cc:5484] T a6b5231a18a5414dbd010d4d48f08dbd P 311f765f59754bdcb4eadc04e14bd4ea reported cstate change: config changed from index 4 to 5, 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "311f765f59754bdcb4eadc04e14bd4ea" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:38.330369 17901 raft_consensus.cc:2932] T a6b5231a18a5414dbd010d4d48f08dbd P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
I20240502 01:54:38.335408 17500 consensus_queue.cc:237] T a6b5231a18a5414dbd010d4d48f08dbd P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } }
I20240502 01:54:38.351547 21427 log.cc:1196] T 37eda67598fe491bbb29aa62b2c4bec7 P 1b50a031f2784ba498b3e52e16916205: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/wal/wals/37eda67598fe491bbb29aa62b2c4bec7
I20240502 01:54:38.352818 17343 catalog_manager.cc:4830] TS 1b50a031f2784ba498b3e52e16916205 (127.9.176.194:45617): tablet 37eda67598fe491bbb29aa62b2c4bec7 (table rebalance_test_table_1 [id=952912d7ec8d493bbb8985fffe9aa918]) successfully deleted
I20240502 01:54:38.360177 21487 ts_tablet_manager.cc:902] T 075ded9523e44c3d92844805338643a3 P 94931f4b51c9493cb3d4264ce99f5e76: Initiating tablet copy from peer 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193:35397)
I20240502 01:54:38.362890 19603 raft_consensus.cc:1261] T a6b5231a18a5414dbd010d4d48f08dbd P 4b7c382c9edd41babdd1a1afc904a813 [term 1 FOLLOWER]: Refusing update from remote peer 311f765f59754bdcb4eadc04e14bd4ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:38.364106 21467 consensus_queue.cc:1035] T a6b5231a18a5414dbd010d4d48f08dbd P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:38.326838 17633 raft_consensus.cc:2932] T a6b5231a18a5414dbd010d4d48f08dbd P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
I20240502 01:54:38.367822 20788 raft_consensus.cc:2932] T a6b5231a18a5414dbd010d4d48f08dbd P 311f765f59754bdcb4eadc04e14bd4ea [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER da3584f429104d23a1a8899b35cc2138 (127.9.176.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
I20240502 01:54:38.374650 17343 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a6b5231a18a5414dbd010d4d48f08dbd with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:38.375993 19603 raft_consensus.cc:2932] T a6b5231a18a5414dbd010d4d48f08dbd P 4b7c382c9edd41babdd1a1afc904a813 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER da3584f429104d23a1a8899b35cc2138 (127.9.176.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
I20240502 01:54:38.382187 17352 catalog_manager.cc:5484] T a6b5231a18a5414dbd010d4d48f08dbd P 311f765f59754bdcb4eadc04e14bd4ea reported cstate change: config changed from index 5 to 6, VOTER da3584f429104d23a1a8899b35cc2138 (127.9.176.196) evicted. New cstate: current_term: 1 leader_uuid: "311f765f59754bdcb4eadc04e14bd4ea" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:38.384474 19467 raft_consensus.cc:1261] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 29806d3d37354c169484f6db368f8798 [term 1 LEARNER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:38.385339 21143 consensus_queue.cc:1035] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:38.387790 17880 tablet_service.cc:1514] Processing DeleteTablet for tablet a6b5231a18a5414dbd010d4d48f08dbd with delete_type TABLET_DATA_TOMBSTONED (TS da3584f429104d23a1a8899b35cc2138 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:34396
I20240502 01:54:38.419338 21241 tablet_bootstrap.cc:492] T 49ef898a43704b87ba077cd7b0485e9a P 94931f4b51c9493cb3d4264ce99f5e76: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:38.419732 17898 raft_consensus.cc:1261] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:38.420740 21143 consensus_queue.cc:1035] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:38.435021 17500 raft_consensus.cc:1261] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:38.436081 21362 consensus_queue.cc:1035] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240502 01:54:38.446480 21241 tablet_bootstrap.cc:492] T 49ef898a43704b87ba077cd7b0485e9a P 94931f4b51c9493cb3d4264ce99f5e76: Bootstrap complete.
I20240502 01:54:38.446919 21241 ts_tablet_manager.cc:1367] T 49ef898a43704b87ba077cd7b0485e9a P 94931f4b51c9493cb3d4264ce99f5e76: Time spent bootstrapping tablet: real 0.509s	user 0.041s	sys 0.017s
I20240502 01:54:38.450134 21241 raft_consensus.cc:348] T 49ef898a43704b87ba077cd7b0485e9a P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } }
I20240502 01:54:38.450517 21241 raft_consensus.cc:729] T 49ef898a43704b87ba077cd7b0485e9a P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 94931f4b51c9493cb3d4264ce99f5e76, State: Initialized, Role: LEARNER
I20240502 01:54:38.452549 17632 raft_consensus.cc:1261] T a6b5231a18a5414dbd010d4d48f08dbd P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Refusing update from remote peer 311f765f59754bdcb4eadc04e14bd4ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:38.458536 20788 consensus_queue.cc:1035] T a6b5231a18a5414dbd010d4d48f08dbd P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:38.469408 17632 raft_consensus.cc:2932] T a6b5231a18a5414dbd010d4d48f08dbd P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER da3584f429104d23a1a8899b35cc2138 (127.9.176.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
I20240502 01:54:38.472905 17613 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:54:38.477054 17631 consensus_queue.cc:237] T 5f3f3e5f772b4566b69236fd87d71a38 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } }
I20240502 01:54:38.484289 21143 raft_consensus.cc:2932] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 84d3a4b174c34e02aedab1e620731359 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 29806d3d37354c169484f6db368f8798 (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } }
I20240502 01:54:38.490460 21241 consensus_queue.cc:260] T 49ef898a43704b87ba077cd7b0485e9a P 94931f4b51c9493cb3d4264ce99f5e76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } }
I20240502 01:54:38.491266 21241 ts_tablet_manager.cc:1398] T 49ef898a43704b87ba077cd7b0485e9a P 94931f4b51c9493cb3d4264ce99f5e76: Time spent starting tablet: real 0.044s	user 0.003s	sys 0.000s
I20240502 01:54:38.493078 21424 tablet_bootstrap.cc:492] T 62fd543d5c7c41f98926f2148ee61e7d P 272b2179c1b4476090a89880df2a8baa: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:38.493577 21424 tablet_bootstrap.cc:492] T 62fd543d5c7c41f98926f2148ee61e7d P 272b2179c1b4476090a89880df2a8baa: Bootstrap complete.
I20240502 01:54:38.493939 21424 ts_tablet_manager.cc:1367] T 62fd543d5c7c41f98926f2148ee61e7d P 272b2179c1b4476090a89880df2a8baa: Time spent bootstrapping tablet: real 0.277s	user 0.060s	sys 0.004s
I20240502 01:54:38.494035 17500 raft_consensus.cc:2932] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 29806d3d37354c169484f6db368f8798 (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } }
I20240502 01:54:38.494952 21424 raft_consensus.cc:348] T 62fd543d5c7c41f98926f2148ee61e7d P 272b2179c1b4476090a89880df2a8baa [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } }
I20240502 01:54:38.495266 21424 raft_consensus.cc:729] T 62fd543d5c7c41f98926f2148ee61e7d P 272b2179c1b4476090a89880df2a8baa [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 272b2179c1b4476090a89880df2a8baa, State: Initialized, Role: LEARNER
I20240502 01:54:38.503813 17746 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:54:38.505605 17898 raft_consensus.cc:2932] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 29806d3d37354c169484f6db368f8798 (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } }
I20240502 01:54:38.506245 17520 tablet_copy_service.cc:342] P 311f765f59754bdcb4eadc04e14bd4ea: Request end of tablet copy session 94931f4b51c9493cb3d4264ce99f5e76-49ef898a43704b87ba077cd7b0485e9a received from {username='slave'} at 127.9.176.200:46111
I20240502 01:54:38.506393 17520 tablet_copy_service.cc:433] P 311f765f59754bdcb4eadc04e14bd4ea: ending tablet copy session 94931f4b51c9493cb3d4264ce99f5e76-49ef898a43704b87ba077cd7b0485e9a on tablet 49ef898a43704b87ba077cd7b0485e9a with peer 94931f4b51c9493cb3d4264ce99f5e76
I20240502 01:54:38.508411 21462 tablet_copy_client.cc:308] T 0dd8944f9fd542e29072ca501df5e112 P 29806d3d37354c169484f6db368f8798: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.194:45617
I20240502 01:54:38.514833 19585 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:38.521602 21505 tablet_replica.cc:330] T a6b5231a18a5414dbd010d4d48f08dbd P da3584f429104d23a1a8899b35cc2138: stopping tablet replica
I20240502 01:54:38.521970 21505 raft_consensus.cc:2229] T a6b5231a18a5414dbd010d4d48f08dbd P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:38.522356 21505 raft_consensus.cc:2258] T a6b5231a18a5414dbd010d4d48f08dbd P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:38.523885 21505 ts_tablet_manager.cc:1875] T a6b5231a18a5414dbd010d4d48f08dbd P da3584f429104d23a1a8899b35cc2138: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:38.511482 17355 catalog_manager.cc:5484] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 84d3a4b174c34e02aedab1e620731359 reported cstate change: config changed from index 4 to 5, 29806d3d37354c169484f6db368f8798 (127.9.176.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "84d3a4b174c34e02aedab1e620731359" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:38.539716 17901 raft_consensus.cc:1261] T 5f3f3e5f772b4566b69236fd87d71a38 P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:38.540485 20628 consensus_queue.cc:1035] T 5f3f3e5f772b4566b69236fd87d71a38 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:38.543936 21487 tablet_copy_client.cc:308] T 075ded9523e44c3d92844805338643a3 P 94931f4b51c9493cb3d4264ce99f5e76: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.193:35397
I20240502 01:54:38.545364 17654 tablet_copy_service.cc:140] P 1b50a031f2784ba498b3e52e16916205: Received BeginTabletCopySession request for tablet 0dd8944f9fd542e29072ca501df5e112 from peer 29806d3d37354c169484f6db368f8798 ({username='slave'} at 127.9.176.197:33725)
I20240502 01:54:38.545598 17654 tablet_copy_service.cc:161] P 1b50a031f2784ba498b3e52e16916205: Beginning new tablet copy session on tablet 0dd8944f9fd542e29072ca501df5e112 from peer 29806d3d37354c169484f6db368f8798 at {username='slave'} at 127.9.176.197:33725: session id = 29806d3d37354c169484f6db368f8798-0dd8944f9fd542e29072ca501df5e112
I20240502 01:54:38.546440 19871 raft_consensus.cc:1203] T 49ef898a43704b87ba077cd7b0485e9a P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 01:54:38.547003 17520 tablet_copy_service.cc:140] P 311f765f59754bdcb4eadc04e14bd4ea: Received BeginTabletCopySession request for tablet 075ded9523e44c3d92844805338643a3 from peer 94931f4b51c9493cb3d4264ce99f5e76 ({username='slave'} at 127.9.176.200:46111)
I20240502 01:54:38.547160 17520 tablet_copy_service.cc:161] P 311f765f59754bdcb4eadc04e14bd4ea: Beginning new tablet copy session on tablet 075ded9523e44c3d92844805338643a3 from peer 94931f4b51c9493cb3d4264ce99f5e76 at {username='slave'} at 127.9.176.200:46111: session id = 94931f4b51c9493cb3d4264ce99f5e76-075ded9523e44c3d92844805338643a3
I20240502 01:54:38.549021 17654 tablet_copy_source_session.cc:205] T 0dd8944f9fd542e29072ca501df5e112 P 1b50a031f2784ba498b3e52e16916205: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:38.550415 17520 tablet_copy_source_session.cc:205] T 075ded9523e44c3d92844805338643a3 P 311f765f59754bdcb4eadc04e14bd4ea: Tablet Copy: opened 0 blocks and 1 log segments
W20240502 01:54:38.551275 17565 consensus_peers.cc:485] T 5f3f3e5f772b4566b69236fd87d71a38 P 1b50a031f2784ba498b3e52e16916205 -> Peer 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198:36865): Couldn't send request to peer 4b7c382c9edd41babdd1a1afc904a813. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5f3f3e5f772b4566b69236fd87d71a38. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:38.561058 21505 ts_tablet_manager.cc:1888] T a6b5231a18a5414dbd010d4d48f08dbd P da3584f429104d23a1a8899b35cc2138: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:54:38.562889 21505 log.cc:1196] T a6b5231a18a5414dbd010d4d48f08dbd P da3584f429104d23a1a8899b35cc2138: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/wal/wals/a6b5231a18a5414dbd010d4d48f08dbd
I20240502 01:54:38.565191 21487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 075ded9523e44c3d92844805338643a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:38.566350 19465 raft_consensus.cc:2932] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 29806d3d37354c169484f6db368f8798 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 29806d3d37354c169484f6db368f8798 (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } }
I20240502 01:54:38.567710 17340 catalog_manager.cc:4830] TS da3584f429104d23a1a8899b35cc2138 (127.9.176.196:46381): tablet a6b5231a18a5414dbd010d4d48f08dbd (table rebalance_test_table_6 [id=d49bf19a92cf45ceadf02d01dbdae6ad]) successfully deleted
I20240502 01:54:38.574514 21462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dd8944f9fd542e29072ca501df5e112. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:38.585973 21487 tablet_copy_client.cc:783] T 075ded9523e44c3d92844805338643a3 P 94931f4b51c9493cb3d4264ce99f5e76: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:38.586357 21487 tablet_copy_client.cc:647] T 075ded9523e44c3d92844805338643a3 P 94931f4b51c9493cb3d4264ce99f5e76: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:38.587478 17767 consensus_queue.cc:237] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } }
I20240502 01:54:38.591140 17763 raft_consensus.cc:1261] T 5f3f3e5f772b4566b69236fd87d71a38 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:38.591786 17500 raft_consensus.cc:1261] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:38.592029 21487 tablet_copy_client.cc:523] T 075ded9523e44c3d92844805338643a3 P 94931f4b51c9493cb3d4264ce99f5e76: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:38.592490 21143 consensus_queue.cc:1035] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:38.592726 20628 consensus_queue.cc:1035] T 5f3f3e5f772b4566b69236fd87d71a38 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:38.595579 21363 raft_consensus.cc:2932] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 84d3a4b174c34e02aedab1e620731359 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER da3584f429104d23a1a8899b35cc2138 (127.9.176.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } }
I20240502 01:54:38.495504 21424 consensus_queue.cc:260] T 62fd543d5c7c41f98926f2148ee61e7d P 272b2179c1b4476090a89880df2a8baa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } }
I20240502 01:54:38.600878 17343 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3ca5c4988ff74e3ca8d7a5614bcb93e2 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:38.602622 21487 tablet_bootstrap.cc:492] T 075ded9523e44c3d92844805338643a3 P 94931f4b51c9493cb3d4264ce99f5e76: Bootstrap starting.
I20240502 01:54:38.604871 17499 raft_consensus.cc:2932] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER da3584f429104d23a1a8899b35cc2138 (127.9.176.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } }
I20240502 01:54:38.605918 19852 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:38.615962 17478 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:54:38.628204 20628 raft_consensus.cc:2932] T 5f3f3e5f772b4566b69236fd87d71a38 P 1b50a031f2784ba498b3e52e16916205 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } } }
I20240502 01:54:38.630985 17764 raft_consensus.cc:2932] T 5f3f3e5f772b4566b69236fd87d71a38 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } } }
I20240502 01:54:38.637815 17348 catalog_manager.cc:5484] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 84d3a4b174c34e02aedab1e620731359 reported cstate change: config changed from index 5 to 6, VOTER da3584f429104d23a1a8899b35cc2138 (127.9.176.196) evicted. New cstate: current_term: 1 leader_uuid: "84d3a4b174c34e02aedab1e620731359" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:38.640455 17349 catalog_manager.cc:5484] T 5f3f3e5f772b4566b69236fd87d71a38 P 1b50a031f2784ba498b3e52e16916205 reported cstate change: config changed from index -1 to 4, NON_VOTER 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) added. New cstate: current_term: 1 leader_uuid: "1b50a031f2784ba498b3e52e16916205" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:38.640523 17897 raft_consensus.cc:2932] T 5f3f3e5f772b4566b69236fd87d71a38 P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } } }
I20240502 01:54:38.646529 17880 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ca5c4988ff74e3ca8d7a5614bcb93e2 with delete_type TABLET_DATA_TOMBSTONED (TS da3584f429104d23a1a8899b35cc2138 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:34396
I20240502 01:54:38.666952 19467 raft_consensus.cc:1261] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 29806d3d37354c169484f6db368f8798 [term 1 FOLLOWER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:38.672564 21362 consensus_queue.cc:1035] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:38.674664 21505 tablet_replica.cc:330] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P da3584f429104d23a1a8899b35cc2138: stopping tablet replica
I20240502 01:54:38.674965 21505 raft_consensus.cc:2229] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:38.675230 21505 raft_consensus.cc:2258] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:38.677958 21505 ts_tablet_manager.cc:1875] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P da3584f429104d23a1a8899b35cc2138: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:38.677728 19467 raft_consensus.cc:2932] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 29806d3d37354c169484f6db368f8798 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER da3584f429104d23a1a8899b35cc2138 (127.9.176.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } }
I20240502 01:54:38.684446 21241 ts_tablet_manager.cc:902] T 91e0e82526c0489a8908e250bf3c851f P 94931f4b51c9493cb3d4264ce99f5e76: Initiating tablet copy from peer 1b50a031f2784ba498b3e52e16916205 (127.9.176.194:45617)
I20240502 01:54:38.687733 21433 tablet_bootstrap.cc:492] T 0dd8944f9fd542e29072ca501df5e112 P 272b2179c1b4476090a89880df2a8baa: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:38.688643 21433 tablet_bootstrap.cc:492] T 0dd8944f9fd542e29072ca501df5e112 P 272b2179c1b4476090a89880df2a8baa: Bootstrap complete.
I20240502 01:54:38.688954 21433 ts_tablet_manager.cc:1367] T 0dd8944f9fd542e29072ca501df5e112 P 272b2179c1b4476090a89880df2a8baa: Time spent bootstrapping tablet: real 0.513s	user 0.050s	sys 0.017s
I20240502 01:54:38.694003 21433 raft_consensus.cc:348] T 0dd8944f9fd542e29072ca501df5e112 P 272b2179c1b4476090a89880df2a8baa [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } }
I20240502 01:54:38.694370 21433 raft_consensus.cc:729] T 0dd8944f9fd542e29072ca501df5e112 P 272b2179c1b4476090a89880df2a8baa [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 272b2179c1b4476090a89880df2a8baa, State: Initialized, Role: LEARNER
I20240502 01:54:38.697835 17880 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:54:38.704846 19446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:38.712949 21505 ts_tablet_manager.cc:1888] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P da3584f429104d23a1a8899b35cc2138: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:54:38.694622 21433 consensus_queue.cc:260] T 0dd8944f9fd542e29072ca501df5e112 P 272b2179c1b4476090a89880df2a8baa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } }
I20240502 01:54:38.720166 21424 ts_tablet_manager.cc:1398] T 62fd543d5c7c41f98926f2148ee61e7d P 272b2179c1b4476090a89880df2a8baa: Time spent starting tablet: real 0.226s	user 0.004s	sys 0.014s
I20240502 01:54:38.721251 17520 tablet_copy_service.cc:342] P 311f765f59754bdcb4eadc04e14bd4ea: Request end of tablet copy session 272b2179c1b4476090a89880df2a8baa-62fd543d5c7c41f98926f2148ee61e7d received from {username='slave'} at 127.9.176.199:51681
I20240502 01:54:38.721419 17520 tablet_copy_service.cc:433] P 311f765f59754bdcb4eadc04e14bd4ea: ending tablet copy session 272b2179c1b4476090a89880df2a8baa-62fd543d5c7c41f98926f2148ee61e7d on tablet 62fd543d5c7c41f98926f2148ee61e7d with peer 272b2179c1b4476090a89880df2a8baa
I20240502 01:54:38.725021 21433 ts_tablet_manager.cc:1398] T 0dd8944f9fd542e29072ca501df5e112 P 272b2179c1b4476090a89880df2a8baa: Time spent starting tablet: real 0.036s	user 0.003s	sys 0.000s
I20240502 01:54:38.726194 17654 tablet_copy_service.cc:342] P 1b50a031f2784ba498b3e52e16916205: Request end of tablet copy session 272b2179c1b4476090a89880df2a8baa-0dd8944f9fd542e29072ca501df5e112 received from {username='slave'} at 127.9.176.199:56005
I20240502 01:54:38.726356 17654 tablet_copy_service.cc:433] P 1b50a031f2784ba498b3e52e16916205: ending tablet copy session 272b2179c1b4476090a89880df2a8baa-0dd8944f9fd542e29072ca501df5e112 on tablet 0dd8944f9fd542e29072ca501df5e112 with peer 272b2179c1b4476090a89880df2a8baa
I20240502 01:54:38.732278 19718 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:38.733803 21505 log.cc:1196] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P da3584f429104d23a1a8899b35cc2138: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/wal/wals/3ca5c4988ff74e3ca8d7a5614bcb93e2
I20240502 01:54:38.736202 17340 catalog_manager.cc:4830] TS da3584f429104d23a1a8899b35cc2138 (127.9.176.196:46381): tablet 3ca5c4988ff74e3ca8d7a5614bcb93e2 (table rebalance_test_table_3 [id=c5e1575e337a4256a90ce5b685b3ff28]) successfully deleted
I20240502 01:54:38.810844 21462 tablet_copy_client.cc:783] T 0dd8944f9fd542e29072ca501df5e112 P 29806d3d37354c169484f6db368f8798: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:38.813793 19736 raft_consensus.cc:1203] T 62fd543d5c7c41f98926f2148ee61e7d P 272b2179c1b4476090a89880df2a8baa [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 01:54:38.817647 21565 ts_tablet_manager.cc:902] T 7ce9d4281c1b46258493feb39e8d9ab2 P 4b7c382c9edd41babdd1a1afc904a813: Initiating tablet copy from peer 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193:35397)
I20240502 01:54:38.821727 21241 tablet_copy_client.cc:308] T 91e0e82526c0489a8908e250bf3c851f P 94931f4b51c9493cb3d4264ce99f5e76: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.194:45617
I20240502 01:54:38.822505 17654 tablet_copy_service.cc:140] P 1b50a031f2784ba498b3e52e16916205: Received BeginTabletCopySession request for tablet 91e0e82526c0489a8908e250bf3c851f from peer 94931f4b51c9493cb3d4264ce99f5e76 ({username='slave'} at 127.9.176.200:37843)
I20240502 01:54:38.822675 17654 tablet_copy_service.cc:161] P 1b50a031f2784ba498b3e52e16916205: Beginning new tablet copy session on tablet 91e0e82526c0489a8908e250bf3c851f from peer 94931f4b51c9493cb3d4264ce99f5e76 at {username='slave'} at 127.9.176.200:37843: session id = 94931f4b51c9493cb3d4264ce99f5e76-91e0e82526c0489a8908e250bf3c851f
I20240502 01:54:38.833165 17654 tablet_copy_source_session.cc:205] T 91e0e82526c0489a8908e250bf3c851f P 1b50a031f2784ba498b3e52e16916205: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:38.835139 21241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91e0e82526c0489a8908e250bf3c851f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:38.872658 21241 tablet_copy_client.cc:783] T 91e0e82526c0489a8908e250bf3c851f P 94931f4b51c9493cb3d4264ce99f5e76: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:38.873203 21241 tablet_copy_client.cc:647] T 91e0e82526c0489a8908e250bf3c851f P 94931f4b51c9493cb3d4264ce99f5e76: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:38.878501 20915 raft_consensus.cc:1050] T 17365abbb6c1430aa278719eb8e4a648 P da3584f429104d23a1a8899b35cc2138: attempting to promote NON_VOTER 4b7c382c9edd41babdd1a1afc904a813 to VOTER
I20240502 01:54:38.879632 20915 consensus_queue.cc:237] T 17365abbb6c1430aa278719eb8e4a648 P da3584f429104d23a1a8899b35cc2138 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } }
I20240502 01:54:38.891654 17633 raft_consensus.cc:1261] T 17365abbb6c1430aa278719eb8e4a648 P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Refusing update from remote peer da3584f429104d23a1a8899b35cc2138: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:38.892701 21375 consensus_queue.cc:1035] T 17365abbb6c1430aa278719eb8e4a648 P da3584f429104d23a1a8899b35cc2138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:38.874444 19852 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:38.901667 19603 raft_consensus.cc:1261] T 17365abbb6c1430aa278719eb8e4a648 P 4b7c382c9edd41babdd1a1afc904a813 [term 1 LEARNER]: Refusing update from remote peer da3584f429104d23a1a8899b35cc2138: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:38.901793 19718 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:38.902630 21375 consensus_queue.cc:1035] T 17365abbb6c1430aa278719eb8e4a648 P da3584f429104d23a1a8899b35cc2138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:38.906396 17763 raft_consensus.cc:1261] T 17365abbb6c1430aa278719eb8e4a648 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Refusing update from remote peer da3584f429104d23a1a8899b35cc2138: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:38.907155 21373 consensus_queue.cc:1035] T 17365abbb6c1430aa278719eb8e4a648 P da3584f429104d23a1a8899b35cc2138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:38.918270 17478 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:54:38.964493 21241 tablet_copy_client.cc:523] T 91e0e82526c0489a8908e250bf3c851f P 94931f4b51c9493cb3d4264ce99f5e76: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:38.969604 21462 tablet_copy_client.cc:647] T 0dd8944f9fd542e29072ca501df5e112 P 29806d3d37354c169484f6db368f8798: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:38.970656 17880 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:54:38.978673 21462 tablet_copy_client.cc:523] T 0dd8944f9fd542e29072ca501df5e112 P 29806d3d37354c169484f6db368f8798: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:38.984344 21462 tablet_bootstrap.cc:492] T 0dd8944f9fd542e29072ca501df5e112 P 29806d3d37354c169484f6db368f8798: Bootstrap starting.
I20240502 01:54:39.019704 17746 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:54:39.020249 20915 raft_consensus.cc:2932] T 17365abbb6c1430aa278719eb8e4a648 P da3584f429104d23a1a8899b35cc2138 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
I20240502 01:54:39.027539 19446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:39.027720 17764 raft_consensus.cc:2932] T 17365abbb6c1430aa278719eb8e4a648 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
I20240502 01:54:39.033107 21241 tablet_bootstrap.cc:492] T 91e0e82526c0489a8908e250bf3c851f P 94931f4b51c9493cb3d4264ce99f5e76: Bootstrap starting.
I20240502 01:54:39.037480 21579 ts_tablet_manager.cc:902] T 5f3f3e5f772b4566b69236fd87d71a38 P 4b7c382c9edd41babdd1a1afc904a813: Initiating tablet copy from peer 1b50a031f2784ba498b3e52e16916205 (127.9.176.194:45617)
I20240502 01:54:39.040944 19736 raft_consensus.cc:1203] T 0dd8944f9fd542e29072ca501df5e112 P 272b2179c1b4476090a89880df2a8baa [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
I20240502 01:54:39.047763 17613 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:54:39.052287 17351 catalog_manager.cc:5484] T 17365abbb6c1430aa278719eb8e4a648 P 84d3a4b174c34e02aedab1e620731359 reported cstate change: config changed from index 4 to 5, 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "da3584f429104d23a1a8899b35cc2138" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
I20240502 01:54:39.052496 19605 raft_consensus.cc:2932] T 17365abbb6c1430aa278719eb8e4a648 P 4b7c382c9edd41babdd1a1afc904a813 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
I20240502 01:54:39.062609 17633 raft_consensus.cc:2932] T 17365abbb6c1430aa278719eb8e4a648 P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
I20240502 01:54:39.075865 17898 consensus_queue.cc:237] T 17365abbb6c1430aa278719eb8e4a648 P da3584f429104d23a1a8899b35cc2138 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } }
I20240502 01:54:39.077365 19585 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:39.080083 17767 raft_consensus.cc:1261] T 17365abbb6c1430aa278719eb8e4a648 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Refusing update from remote peer da3584f429104d23a1a8899b35cc2138: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:39.081014 21373 consensus_queue.cc:1035] T 17365abbb6c1430aa278719eb8e4a648 P da3584f429104d23a1a8899b35cc2138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:39.083038 19603 raft_consensus.cc:1261] T 17365abbb6c1430aa278719eb8e4a648 P 4b7c382c9edd41babdd1a1afc904a813 [term 1 FOLLOWER]: Refusing update from remote peer da3584f429104d23a1a8899b35cc2138: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:39.084053 21373 consensus_queue.cc:1035] T 17365abbb6c1430aa278719eb8e4a648 P da3584f429104d23a1a8899b35cc2138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:39.084570 21373 raft_consensus.cc:2932] T 17365abbb6c1430aa278719eb8e4a648 P da3584f429104d23a1a8899b35cc2138 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1b50a031f2784ba498b3e52e16916205 (127.9.176.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
I20240502 01:54:39.089509 21467 raft_consensus.cc:1050] T 49ef898a43704b87ba077cd7b0485e9a P 311f765f59754bdcb4eadc04e14bd4ea: attempting to promote NON_VOTER 94931f4b51c9493cb3d4264ce99f5e76 to VOTER
I20240502 01:54:39.089699 19604 raft_consensus.cc:2932] T 17365abbb6c1430aa278719eb8e4a648 P 4b7c382c9edd41babdd1a1afc904a813 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1b50a031f2784ba498b3e52e16916205 (127.9.176.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
I20240502 01:54:39.090729 21467 consensus_queue.cc:237] T 49ef898a43704b87ba077cd7b0485e9a P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } }
I20240502 01:54:39.108727 19871 raft_consensus.cc:1261] T 49ef898a43704b87ba077cd7b0485e9a P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 LEARNER]: Refusing update from remote peer 311f765f59754bdcb4eadc04e14bd4ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:39.110567 21565 tablet_copy_client.cc:308] T 7ce9d4281c1b46258493feb39e8d9ab2 P 4b7c382c9edd41babdd1a1afc904a813: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.193:35397
I20240502 01:54:39.110875 21409 consensus_queue.cc:1035] T 49ef898a43704b87ba077cd7b0485e9a P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:39.112057 17520 tablet_copy_service.cc:140] P 311f765f59754bdcb4eadc04e14bd4ea: Received BeginTabletCopySession request for tablet 7ce9d4281c1b46258493feb39e8d9ab2 from peer 4b7c382c9edd41babdd1a1afc904a813 ({username='slave'} at 127.9.176.198:53399)
I20240502 01:54:39.112223 17520 tablet_copy_service.cc:161] P 311f765f59754bdcb4eadc04e14bd4ea: Beginning new tablet copy session on tablet 7ce9d4281c1b46258493feb39e8d9ab2 from peer 4b7c382c9edd41babdd1a1afc904a813 at {username='slave'} at 127.9.176.198:53399: session id = 4b7c382c9edd41babdd1a1afc904a813-7ce9d4281c1b46258493feb39e8d9ab2
I20240502 01:54:39.118461 17520 tablet_copy_source_session.cc:205] T 7ce9d4281c1b46258493feb39e8d9ab2 P 311f765f59754bdcb4eadc04e14bd4ea: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:39.119406 17340 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 17365abbb6c1430aa278719eb8e4a648 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:39.121629 21565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ce9d4281c1b46258493feb39e8d9ab2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:39.123857 17631 raft_consensus.cc:1261] T 49ef898a43704b87ba077cd7b0485e9a P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Refusing update from remote peer 311f765f59754bdcb4eadc04e14bd4ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:39.125053 17898 raft_consensus.cc:1261] T 49ef898a43704b87ba077cd7b0485e9a P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Refusing update from remote peer 311f765f59754bdcb4eadc04e14bd4ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:39.125017 21409 consensus_queue.cc:1035] T 49ef898a43704b87ba077cd7b0485e9a P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:39.127264 21342 consensus_queue.cc:1035] T 49ef898a43704b87ba077cd7b0485e9a P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:39.132118 17351 catalog_manager.cc:5484] T 17365abbb6c1430aa278719eb8e4a648 P 4b7c382c9edd41babdd1a1afc904a813 reported cstate change: config changed from index 5 to 6, VOTER 1b50a031f2784ba498b3e52e16916205 (127.9.176.194) evicted. New cstate: current_term: 1 leader_uuid: "da3584f429104d23a1a8899b35cc2138" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
I20240502 01:54:39.141013 17613 tablet_service.cc:1514] Processing DeleteTablet for tablet 17365abbb6c1430aa278719eb8e4a648 with delete_type TABLET_DATA_TOMBSTONED (TS 1b50a031f2784ba498b3e52e16916205 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41484
I20240502 01:54:39.160097 21565 tablet_copy_client.cc:783] T 7ce9d4281c1b46258493feb39e8d9ab2 P 4b7c382c9edd41babdd1a1afc904a813: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:39.160548 21565 tablet_copy_client.cc:647] T 7ce9d4281c1b46258493feb39e8d9ab2 P 4b7c382c9edd41babdd1a1afc904a813: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:39.161342 21592 tablet_replica.cc:330] T 17365abbb6c1430aa278719eb8e4a648 P 1b50a031f2784ba498b3e52e16916205: stopping tablet replica
I20240502 01:54:39.161675 21592 raft_consensus.cc:2229] T 17365abbb6c1430aa278719eb8e4a648 P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:39.162101 21592 raft_consensus.cc:2258] T 17365abbb6c1430aa278719eb8e4a648 P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:39.164072 21592 ts_tablet_manager.cc:1875] T 17365abbb6c1430aa278719eb8e4a648 P 1b50a031f2784ba498b3e52e16916205: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:39.174539 21565 tablet_copy_client.cc:523] T 7ce9d4281c1b46258493feb39e8d9ab2 P 4b7c382c9edd41babdd1a1afc904a813: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:39.179716 17613 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:54:39.104555 17763 raft_consensus.cc:2932] T 17365abbb6c1430aa278719eb8e4a648 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1b50a031f2784ba498b3e52e16916205 (127.9.176.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
I20240502 01:54:39.187769 21565 tablet_bootstrap.cc:492] T 7ce9d4281c1b46258493feb39e8d9ab2 P 4b7c382c9edd41babdd1a1afc904a813: Bootstrap starting.
I20240502 01:54:39.202822 21409 raft_consensus.cc:2932] T 49ef898a43704b87ba077cd7b0485e9a P 311f765f59754bdcb4eadc04e14bd4ea [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
I20240502 01:54:39.209456 17631 raft_consensus.cc:2932] T 49ef898a43704b87ba077cd7b0485e9a P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
I20240502 01:54:39.213142 17354 catalog_manager.cc:5484] T 49ef898a43704b87ba077cd7b0485e9a P 311f765f59754bdcb4eadc04e14bd4ea reported cstate change: config changed from index 4 to 5, 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "311f765f59754bdcb4eadc04e14bd4ea" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:39.214869 17898 raft_consensus.cc:2932] T 49ef898a43704b87ba077cd7b0485e9a P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
I20240502 01:54:39.219619 21241 tablet_bootstrap.cc:700] T 91e0e82526c0489a8908e250bf3c851f P 94931f4b51c9493cb3d4264ce99f5e76: Time spent opening tablet: real 0.149s	user 0.002s	sys 0.001s
I20240502 01:54:39.235473 19718 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:39.235750 19871 raft_consensus.cc:2932] T 49ef898a43704b87ba077cd7b0485e9a P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
I20240502 01:54:39.260931 17478 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:54:39.283651 21579 tablet_copy_client.cc:308] T 5f3f3e5f772b4566b69236fd87d71a38 P 4b7c382c9edd41babdd1a1afc904a813: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.194:45617
I20240502 01:54:39.284570 17654 tablet_copy_service.cc:140] P 1b50a031f2784ba498b3e52e16916205: Received BeginTabletCopySession request for tablet 5f3f3e5f772b4566b69236fd87d71a38 from peer 4b7c382c9edd41babdd1a1afc904a813 ({username='slave'} at 127.9.176.198:49387)
I20240502 01:54:39.284792 17654 tablet_copy_service.cc:161] P 1b50a031f2784ba498b3e52e16916205: Beginning new tablet copy session on tablet 5f3f3e5f772b4566b69236fd87d71a38 from peer 4b7c382c9edd41babdd1a1afc904a813 at {username='slave'} at 127.9.176.198:49387: session id = 4b7c382c9edd41babdd1a1afc904a813-5f3f3e5f772b4566b69236fd87d71a38
I20240502 01:54:39.290217 17654 tablet_copy_source_session.cc:205] T 5f3f3e5f772b4566b69236fd87d71a38 P 1b50a031f2784ba498b3e52e16916205: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:39.291455 19446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:39.291841 21342 raft_consensus.cc:1050] T 62fd543d5c7c41f98926f2148ee61e7d P 311f765f59754bdcb4eadc04e14bd4ea: attempting to promote NON_VOTER 272b2179c1b4476090a89880df2a8baa to VOTER
I20240502 01:54:39.292738 21342 consensus_queue.cc:237] T 62fd543d5c7c41f98926f2148ee61e7d P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } }
I20240502 01:54:39.295861 21579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f3f3e5f772b4566b69236fd87d71a38. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:39.257853 21592 ts_tablet_manager.cc:1888] T 17365abbb6c1430aa278719eb8e4a648 P 1b50a031f2784ba498b3e52e16916205: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:54:39.308399 21592 log.cc:1196] T 17365abbb6c1430aa278719eb8e4a648 P 1b50a031f2784ba498b3e52e16916205: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/wal/wals/17365abbb6c1430aa278719eb8e4a648
I20240502 01:54:39.310791 17343 catalog_manager.cc:4830] TS 1b50a031f2784ba498b3e52e16916205 (127.9.176.194:45617): tablet 17365abbb6c1430aa278719eb8e4a648 (table rebalance_test_table_5 [id=73ba62f6c85943bf85dfaaf459d924f0]) successfully deleted
I20240502 01:54:39.325497 19736 raft_consensus.cc:1261] T 62fd543d5c7c41f98926f2148ee61e7d P 272b2179c1b4476090a89880df2a8baa [term 1 LEARNER]: Refusing update from remote peer 311f765f59754bdcb4eadc04e14bd4ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:39.325568 17633 raft_consensus.cc:1261] T 62fd543d5c7c41f98926f2148ee61e7d P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Refusing update from remote peer 311f765f59754bdcb4eadc04e14bd4ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:39.326207 20788 consensus_queue.cc:1035] T 62fd543d5c7c41f98926f2148ee61e7d P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:39.326944 21409 consensus_queue.cc:1035] T 62fd543d5c7c41f98926f2148ee61e7d P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:39.328227 21212 tablet_bootstrap.cc:492] T a5e03cf44eab41cc842342a138c04067 P 29806d3d37354c169484f6db368f8798: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:39.328709 21212 tablet_bootstrap.cc:492] T a5e03cf44eab41cc842342a138c04067 P 29806d3d37354c169484f6db368f8798: Bootstrap complete.
I20240502 01:54:39.332849 17763 raft_consensus.cc:1261] T 62fd543d5c7c41f98926f2148ee61e7d P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Refusing update from remote peer 311f765f59754bdcb4eadc04e14bd4ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:39.333559 21342 consensus_queue.cc:1035] T 62fd543d5c7c41f98926f2148ee61e7d P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:39.350253 21409 raft_consensus.cc:2932] T 62fd543d5c7c41f98926f2148ee61e7d P 311f765f59754bdcb4eadc04e14bd4ea [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 272b2179c1b4476090a89880df2a8baa (127.9.176.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } }
I20240502 01:54:39.351819 17763 raft_consensus.cc:2932] T 62fd543d5c7c41f98926f2148ee61e7d P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 272b2179c1b4476090a89880df2a8baa (127.9.176.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } }
I20240502 01:54:39.355446 19736 raft_consensus.cc:2932] T 62fd543d5c7c41f98926f2148ee61e7d P 272b2179c1b4476090a89880df2a8baa [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 272b2179c1b4476090a89880df2a8baa (127.9.176.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } }
I20240502 01:54:39.358141 17746 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:54:39.358330 17349 catalog_manager.cc:5484] T 62fd543d5c7c41f98926f2148ee61e7d P 311f765f59754bdcb4eadc04e14bd4ea reported cstate change: config changed from index 4 to 5, 272b2179c1b4476090a89880df2a8baa (127.9.176.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "311f765f59754bdcb4eadc04e14bd4ea" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:39.366911 17500 consensus_queue.cc:237] T 62fd543d5c7c41f98926f2148ee61e7d P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } }
I20240502 01:54:39.374327 17633 raft_consensus.cc:2932] T 62fd543d5c7c41f98926f2148ee61e7d P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 272b2179c1b4476090a89880df2a8baa (127.9.176.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } }
I20240502 01:54:39.387724 21212 ts_tablet_manager.cc:1367] T a5e03cf44eab41cc842342a138c04067 P 29806d3d37354c169484f6db368f8798: Time spent bootstrapping tablet: real 1.289s	user 0.054s	sys 0.005s
I20240502 01:54:39.388724 21212 raft_consensus.cc:348] T a5e03cf44eab41cc842342a138c04067 P 29806d3d37354c169484f6db368f8798 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } }
I20240502 01:54:39.389423 21212 raft_consensus.cc:729] T a5e03cf44eab41cc842342a138c04067 P 29806d3d37354c169484f6db368f8798 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 29806d3d37354c169484f6db368f8798, State: Initialized, Role: LEARNER
I20240502 01:54:39.389707 21212 consensus_queue.cc:260] T a5e03cf44eab41cc842342a138c04067 P 29806d3d37354c169484f6db368f8798 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } }
I20240502 01:54:39.390129 19736 raft_consensus.cc:1261] T 62fd543d5c7c41f98926f2148ee61e7d P 272b2179c1b4476090a89880df2a8baa [term 1 FOLLOWER]: Refusing update from remote peer 311f765f59754bdcb4eadc04e14bd4ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:39.392915 20788 consensus_queue.cc:1035] T 62fd543d5c7c41f98926f2148ee61e7d P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:39.400830 17880 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:54:39.411419 21579 tablet_copy_client.cc:783] T 5f3f3e5f772b4566b69236fd87d71a38 P 4b7c382c9edd41babdd1a1afc904a813: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:39.411870 21579 tablet_copy_client.cc:647] T 5f3f3e5f772b4566b69236fd87d71a38 P 4b7c382c9edd41babdd1a1afc904a813: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:39.415868 17764 raft_consensus.cc:1261] T 62fd543d5c7c41f98926f2148ee61e7d P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Refusing update from remote peer 311f765f59754bdcb4eadc04e14bd4ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:39.416411 21579 tablet_copy_client.cc:523] T 5f3f3e5f772b4566b69236fd87d71a38 P 4b7c382c9edd41babdd1a1afc904a813: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:39.417834 20788 consensus_queue.cc:1035] T 62fd543d5c7c41f98926f2148ee61e7d P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:39.418655 21409 raft_consensus.cc:2932] T 62fd543d5c7c41f98926f2148ee61e7d P 311f765f59754bdcb4eadc04e14bd4ea [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1b50a031f2784ba498b3e52e16916205 (127.9.176.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } }
I20240502 01:54:39.423002 17343 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 62fd543d5c7c41f98926f2148ee61e7d with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:39.424122 19736 raft_consensus.cc:2932] T 62fd543d5c7c41f98926f2148ee61e7d P 272b2179c1b4476090a89880df2a8baa [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1b50a031f2784ba498b3e52e16916205 (127.9.176.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } }
I20240502 01:54:39.427913 17764 raft_consensus.cc:2932] T 62fd543d5c7c41f98926f2148ee61e7d P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1b50a031f2784ba498b3e52e16916205 (127.9.176.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } }
I20240502 01:54:39.429771 21212 ts_tablet_manager.cc:1398] T a5e03cf44eab41cc842342a138c04067 P 29806d3d37354c169484f6db368f8798: Time spent starting tablet: real 0.042s	user 0.002s	sys 0.000s
I20240502 01:54:39.430229 17349 catalog_manager.cc:5484] T 62fd543d5c7c41f98926f2148ee61e7d P 311f765f59754bdcb4eadc04e14bd4ea reported cstate change: config changed from index 5 to 6, VOTER 1b50a031f2784ba498b3e52e16916205 (127.9.176.194) evicted. New cstate: current_term: 1 leader_uuid: "311f765f59754bdcb4eadc04e14bd4ea" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:39.432507 17922 tablet_copy_service.cc:342] P da3584f429104d23a1a8899b35cc2138: Request end of tablet copy session 29806d3d37354c169484f6db368f8798-a5e03cf44eab41cc842342a138c04067 received from {username='slave'} at 127.9.176.197:44989
I20240502 01:54:39.432660 17922 tablet_copy_service.cc:433] P da3584f429104d23a1a8899b35cc2138: ending tablet copy session 29806d3d37354c169484f6db368f8798-a5e03cf44eab41cc842342a138c04067 on tablet a5e03cf44eab41cc842342a138c04067 with peer 29806d3d37354c169484f6db368f8798
I20240502 01:54:39.437279 17613 tablet_service.cc:1514] Processing DeleteTablet for tablet 62fd543d5c7c41f98926f2148ee61e7d with delete_type TABLET_DATA_TOMBSTONED (TS 1b50a031f2784ba498b3e52e16916205 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41484
I20240502 01:54:39.437747 21592 tablet_replica.cc:330] T 62fd543d5c7c41f98926f2148ee61e7d P 1b50a031f2784ba498b3e52e16916205: stopping tablet replica
I20240502 01:54:39.438021 21592 raft_consensus.cc:2229] T 62fd543d5c7c41f98926f2148ee61e7d P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:39.438318 21592 raft_consensus.cc:2258] T 62fd543d5c7c41f98926f2148ee61e7d P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:39.439846 21592 ts_tablet_manager.cc:1875] T 62fd543d5c7c41f98926f2148ee61e7d P 1b50a031f2784ba498b3e52e16916205: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:39.439951 19852 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:39.447467 21579 tablet_bootstrap.cc:492] T 5f3f3e5f772b4566b69236fd87d71a38 P 4b7c382c9edd41babdd1a1afc904a813: Bootstrap starting.
I20240502 01:54:39.450835 19585 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:39.487373 20913 raft_consensus.cc:1050] T 0dd8944f9fd542e29072ca501df5e112 P 1b50a031f2784ba498b3e52e16916205: attempting to promote NON_VOTER 272b2179c1b4476090a89880df2a8baa to VOTER
I20240502 01:54:39.489061 20913 consensus_queue.cc:237] T 0dd8944f9fd542e29072ca501df5e112 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } }
W20240502 01:54:39.507723 17565 consensus_peers.cc:485] T 0dd8944f9fd542e29072ca501df5e112 P 1b50a031f2784ba498b3e52e16916205 -> Peer 29806d3d37354c169484f6db368f8798 (127.9.176.197:35011): Couldn't send request to peer 29806d3d37354c169484f6db368f8798. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:39.511257 17763 raft_consensus.cc:1261] T 0dd8944f9fd542e29072ca501df5e112 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:39.511999 19738 raft_consensus.cc:1261] T 0dd8944f9fd542e29072ca501df5e112 P 272b2179c1b4476090a89880df2a8baa [term 1 LEARNER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:39.513046 20628 consensus_queue.cc:1035] T 0dd8944f9fd542e29072ca501df5e112 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:39.513540 20628 consensus_queue.cc:1035] T 0dd8944f9fd542e29072ca501df5e112 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:39.521350 21462 tablet_bootstrap.cc:492] T 0dd8944f9fd542e29072ca501df5e112 P 29806d3d37354c169484f6db368f8798: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:39.521905 21462 tablet_bootstrap.cc:492] T 0dd8944f9fd542e29072ca501df5e112 P 29806d3d37354c169484f6db368f8798: Bootstrap complete.
I20240502 01:54:39.522231 21462 ts_tablet_manager.cc:1367] T 0dd8944f9fd542e29072ca501df5e112 P 29806d3d37354c169484f6db368f8798: Time spent bootstrapping tablet: real 0.539s	user 0.055s	sys 0.004s
I20240502 01:54:39.525511 21462 raft_consensus.cc:348] T 0dd8944f9fd542e29072ca501df5e112 P 29806d3d37354c169484f6db368f8798 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } }
I20240502 01:54:39.525838 21462 raft_consensus.cc:729] T 0dd8944f9fd542e29072ca501df5e112 P 29806d3d37354c169484f6db368f8798 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 29806d3d37354c169484f6db368f8798, State: Initialized, Role: LEARNER
I20240502 01:54:39.526064 21462 consensus_queue.cc:260] T 0dd8944f9fd542e29072ca501df5e112 P 29806d3d37354c169484f6db368f8798 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } }
I20240502 01:54:39.526922 21462 ts_tablet_manager.cc:1398] T 0dd8944f9fd542e29072ca501df5e112 P 29806d3d37354c169484f6db368f8798: Time spent starting tablet: real 0.005s	user 0.005s	sys 0.000s
I20240502 01:54:39.537454 17654 tablet_copy_service.cc:342] P 1b50a031f2784ba498b3e52e16916205: Request end of tablet copy session 29806d3d37354c169484f6db368f8798-0dd8944f9fd542e29072ca501df5e112 received from {username='slave'} at 127.9.176.197:33725
I20240502 01:54:39.537631 17654 tablet_copy_service.cc:433] P 1b50a031f2784ba498b3e52e16916205: ending tablet copy session 29806d3d37354c169484f6db368f8798-0dd8944f9fd542e29072ca501df5e112 on tablet 0dd8944f9fd542e29072ca501df5e112 with peer 29806d3d37354c169484f6db368f8798
I20240502 01:54:39.544642 21592 ts_tablet_manager.cc:1888] T 62fd543d5c7c41f98926f2148ee61e7d P 1b50a031f2784ba498b3e52e16916205: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:54:39.544901 21592 log.cc:1196] T 62fd543d5c7c41f98926f2148ee61e7d P 1b50a031f2784ba498b3e52e16916205: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/wal/wals/62fd543d5c7c41f98926f2148ee61e7d
I20240502 01:54:39.545886 17343 catalog_manager.cc:4830] TS 1b50a031f2784ba498b3e52e16916205 (127.9.176.194:45617): tablet 62fd543d5c7c41f98926f2148ee61e7d (table rebalance_test_table_0 [id=c516e7439e734f02a0947c3680cf79e2]) successfully deleted
I20240502 01:54:39.624485 19738 raft_consensus.cc:2932] T 0dd8944f9fd542e29072ca501df5e112 P 272b2179c1b4476090a89880df2a8baa [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 272b2179c1b4476090a89880df2a8baa (127.9.176.199) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } } }
I20240502 01:54:39.624452 21243 raft_consensus.cc:2932] T 0dd8944f9fd542e29072ca501df5e112 P 1b50a031f2784ba498b3e52e16916205 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 272b2179c1b4476090a89880df2a8baa (127.9.176.199) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } } }
I20240502 01:54:39.630825 17354 catalog_manager.cc:5484] T 0dd8944f9fd542e29072ca501df5e112 P 272b2179c1b4476090a89880df2a8baa reported cstate change: config changed from index 5 to 6, 272b2179c1b4476090a89880df2a8baa (127.9.176.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1b50a031f2784ba498b3e52e16916205" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } } }
I20240502 01:54:39.631875 17500 raft_consensus.cc:1261] T 0dd8944f9fd542e29072ca501df5e112 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:39.632625 20628 consensus_queue.cc:1035] T 0dd8944f9fd542e29072ca501df5e112 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:39.638360 21565 tablet_bootstrap.cc:492] T 7ce9d4281c1b46258493feb39e8d9ab2 P 4b7c382c9edd41babdd1a1afc904a813: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:39.639043 21565 tablet_bootstrap.cc:492] T 7ce9d4281c1b46258493feb39e8d9ab2 P 4b7c382c9edd41babdd1a1afc904a813: Bootstrap complete.
I20240502 01:54:39.639410 21565 ts_tablet_manager.cc:1367] T 7ce9d4281c1b46258493feb39e8d9ab2 P 4b7c382c9edd41babdd1a1afc904a813: Time spent bootstrapping tablet: real 0.461s	user 0.052s	sys 0.007s
I20240502 01:54:39.640357 21565 raft_consensus.cc:348] T 7ce9d4281c1b46258493feb39e8d9ab2 P 4b7c382c9edd41babdd1a1afc904a813 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } }
I20240502 01:54:39.640779 21565 raft_consensus.cc:729] T 7ce9d4281c1b46258493feb39e8d9ab2 P 4b7c382c9edd41babdd1a1afc904a813 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4b7c382c9edd41babdd1a1afc904a813, State: Initialized, Role: LEARNER
I20240502 01:54:39.641150 21565 consensus_queue.cc:260] T 7ce9d4281c1b46258493feb39e8d9ab2 P 4b7c382c9edd41babdd1a1afc904a813 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } }
I20240502 01:54:39.660356 17500 raft_consensus.cc:2932] T 0dd8944f9fd542e29072ca501df5e112 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 272b2179c1b4476090a89880df2a8baa (127.9.176.199) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } } }
I20240502 01:54:39.666368 17763 raft_consensus.cc:2932] T 0dd8944f9fd542e29072ca501df5e112 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 272b2179c1b4476090a89880df2a8baa (127.9.176.199) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } } }
I20240502 01:54:39.700939 21487 tablet_bootstrap.cc:492] T 075ded9523e44c3d92844805338643a3 P 94931f4b51c9493cb3d4264ce99f5e76: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:39.701534 21487 tablet_bootstrap.cc:492] T 075ded9523e44c3d92844805338643a3 P 94931f4b51c9493cb3d4264ce99f5e76: Bootstrap complete.
I20240502 01:54:39.702085 21487 ts_tablet_manager.cc:1367] T 075ded9523e44c3d92844805338643a3 P 94931f4b51c9493cb3d4264ce99f5e76: Time spent bootstrapping tablet: real 1.100s	user 0.055s	sys 0.011s
I20240502 01:54:39.703369 21487 raft_consensus.cc:348] T 075ded9523e44c3d92844805338643a3 P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } }
I20240502 01:54:39.703796 21487 raft_consensus.cc:729] T 075ded9523e44c3d92844805338643a3 P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 94931f4b51c9493cb3d4264ce99f5e76, State: Initialized, Role: LEARNER
I20240502 01:54:39.704110 21487 consensus_queue.cc:260] T 075ded9523e44c3d92844805338643a3 P 94931f4b51c9493cb3d4264ce99f5e76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } }
I20240502 01:54:39.718796 21579 tablet_bootstrap.cc:492] T 5f3f3e5f772b4566b69236fd87d71a38 P 4b7c382c9edd41babdd1a1afc904a813: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:39.719328 21579 tablet_bootstrap.cc:492] T 5f3f3e5f772b4566b69236fd87d71a38 P 4b7c382c9edd41babdd1a1afc904a813: Bootstrap complete.
I20240502 01:54:39.719683 21579 ts_tablet_manager.cc:1367] T 5f3f3e5f772b4566b69236fd87d71a38 P 4b7c382c9edd41babdd1a1afc904a813: Time spent bootstrapping tablet: real 0.272s	user 0.056s	sys 0.015s
I20240502 01:54:39.733206 21579 raft_consensus.cc:348] T 5f3f3e5f772b4566b69236fd87d71a38 P 4b7c382c9edd41babdd1a1afc904a813 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } }
I20240502 01:54:39.733610 21579 raft_consensus.cc:729] T 5f3f3e5f772b4566b69236fd87d71a38 P 4b7c382c9edd41babdd1a1afc904a813 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4b7c382c9edd41babdd1a1afc904a813, State: Initialized, Role: LEARNER
I20240502 01:54:39.733892 21579 consensus_queue.cc:260] T 5f3f3e5f772b4566b69236fd87d71a38 P 4b7c382c9edd41babdd1a1afc904a813 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } }
I20240502 01:54:39.752424 21487 ts_tablet_manager.cc:1398] T 075ded9523e44c3d92844805338643a3 P 94931f4b51c9493cb3d4264ce99f5e76: Time spent starting tablet: real 0.050s	user 0.002s	sys 0.000s
I20240502 01:54:39.753579 17520 tablet_copy_service.cc:342] P 311f765f59754bdcb4eadc04e14bd4ea: Request end of tablet copy session 94931f4b51c9493cb3d4264ce99f5e76-075ded9523e44c3d92844805338643a3 received from {username='slave'} at 127.9.176.200:46111
I20240502 01:54:39.753808 17520 tablet_copy_service.cc:433] P 311f765f59754bdcb4eadc04e14bd4ea: ending tablet copy session 94931f4b51c9493cb3d4264ce99f5e76-075ded9523e44c3d92844805338643a3 on tablet 075ded9523e44c3d92844805338643a3 with peer 94931f4b51c9493cb3d4264ce99f5e76
I20240502 01:54:39.758432 21579 ts_tablet_manager.cc:1398] T 5f3f3e5f772b4566b69236fd87d71a38 P 4b7c382c9edd41babdd1a1afc904a813: Time spent starting tablet: real 0.039s	user 0.008s	sys 0.004s
I20240502 01:54:39.760643 19871 raft_consensus.cc:1203] T 075ded9523e44c3d92844805338643a3 P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 01:54:39.761771 17654 tablet_copy_service.cc:342] P 1b50a031f2784ba498b3e52e16916205: Request end of tablet copy session 4b7c382c9edd41babdd1a1afc904a813-5f3f3e5f772b4566b69236fd87d71a38 received from {username='slave'} at 127.9.176.198:49387
I20240502 01:54:39.761956 17654 tablet_copy_service.cc:433] P 1b50a031f2784ba498b3e52e16916205: ending tablet copy session 4b7c382c9edd41babdd1a1afc904a813-5f3f3e5f772b4566b69236fd87d71a38 on tablet 5f3f3e5f772b4566b69236fd87d71a38 with peer 4b7c382c9edd41babdd1a1afc904a813
I20240502 01:54:39.775810 21565 ts_tablet_manager.cc:1398] T 7ce9d4281c1b46258493feb39e8d9ab2 P 4b7c382c9edd41babdd1a1afc904a813: Time spent starting tablet: real 0.136s	user 0.038s	sys 0.080s
I20240502 01:54:39.777683 17520 tablet_copy_service.cc:342] P 311f765f59754bdcb4eadc04e14bd4ea: Request end of tablet copy session 4b7c382c9edd41babdd1a1afc904a813-7ce9d4281c1b46258493feb39e8d9ab2 received from {username='slave'} at 127.9.176.198:53399
I20240502 01:54:39.777869 17520 tablet_copy_service.cc:433] P 311f765f59754bdcb4eadc04e14bd4ea: ending tablet copy session 4b7c382c9edd41babdd1a1afc904a813-7ce9d4281c1b46258493feb39e8d9ab2 on tablet 7ce9d4281c1b46258493feb39e8d9ab2 with peer 4b7c382c9edd41babdd1a1afc904a813
I20240502 01:54:39.797466 19603 raft_consensus.cc:1203] T 7ce9d4281c1b46258493feb39e8d9ab2 P 4b7c382c9edd41babdd1a1afc904a813 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 01:54:39.806767 17763 consensus_queue.cc:237] T c91f74f46b04425aa5393515cb9f781c P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } }
W20240502 01:54:39.844743 17700 consensus_peers.cc:485] T c91f74f46b04425aa5393515cb9f781c P 84d3a4b174c34e02aedab1e620731359 -> Peer 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198:36865): Couldn't send request to peer 4b7c382c9edd41babdd1a1afc904a813. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c91f74f46b04425aa5393515cb9f781c. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:39.848956 19467 raft_consensus.cc:1261] T c91f74f46b04425aa5393515cb9f781c P 29806d3d37354c169484f6db368f8798 [term 1 FOLLOWER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:39.849793 21363 consensus_queue.cc:1035] T c91f74f46b04425aa5393515cb9f781c P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:39.871124 17500 raft_consensus.cc:1261] T c91f74f46b04425aa5393515cb9f781c P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:39.872823 21363 consensus_queue.cc:1035] T c91f74f46b04425aa5393515cb9f781c P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:39.906462 21241 tablet_bootstrap.cc:492] T 91e0e82526c0489a8908e250bf3c851f P 94931f4b51c9493cb3d4264ce99f5e76: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:39.906461 21363 raft_consensus.cc:2932] T c91f74f46b04425aa5393515cb9f781c P 84d3a4b174c34e02aedab1e620731359 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } } }
I20240502 01:54:39.906976 21241 tablet_bootstrap.cc:492] T 91e0e82526c0489a8908e250bf3c851f P 94931f4b51c9493cb3d4264ce99f5e76: Bootstrap complete.
I20240502 01:54:39.907253 21241 ts_tablet_manager.cc:1367] T 91e0e82526c0489a8908e250bf3c851f P 94931f4b51c9493cb3d4264ce99f5e76: Time spent bootstrapping tablet: real 0.874s	user 0.058s	sys 0.003s
I20240502 01:54:39.908180 21241 raft_consensus.cc:348] T 91e0e82526c0489a8908e250bf3c851f P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } }
I20240502 01:54:39.908500 21241 raft_consensus.cc:729] T 91e0e82526c0489a8908e250bf3c851f P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 94931f4b51c9493cb3d4264ce99f5e76, State: Initialized, Role: LEARNER
I20240502 01:54:39.908751 21241 consensus_queue.cc:260] T 91e0e82526c0489a8908e250bf3c851f P 94931f4b51c9493cb3d4264ce99f5e76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } }
I20240502 01:54:39.909334 17499 raft_consensus.cc:2932] T c91f74f46b04425aa5393515cb9f781c P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } } }
I20240502 01:54:39.914958 21241 ts_tablet_manager.cc:1398] T 91e0e82526c0489a8908e250bf3c851f P 94931f4b51c9493cb3d4264ce99f5e76: Time spent starting tablet: real 0.008s	user 0.003s	sys 0.000s
I20240502 01:54:39.919771 17654 tablet_copy_service.cc:342] P 1b50a031f2784ba498b3e52e16916205: Request end of tablet copy session 94931f4b51c9493cb3d4264ce99f5e76-91e0e82526c0489a8908e250bf3c851f received from {username='slave'} at 127.9.176.200:37843
I20240502 01:54:39.919935 17654 tablet_copy_service.cc:433] P 1b50a031f2784ba498b3e52e16916205: ending tablet copy session 94931f4b51c9493cb3d4264ce99f5e76-91e0e82526c0489a8908e250bf3c851f on tablet 91e0e82526c0489a8908e250bf3c851f with peer 94931f4b51c9493cb3d4264ce99f5e76
I20240502 01:54:39.924064 17354 catalog_manager.cc:5484] T c91f74f46b04425aa5393515cb9f781c P 311f765f59754bdcb4eadc04e14bd4ea reported cstate change: config changed from index 6 to 7, NON_VOTER 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) added. New cstate: current_term: 1 leader_uuid: "84d3a4b174c34e02aedab1e620731359" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } } }
I20240502 01:54:39.941882 19465 raft_consensus.cc:2932] T c91f74f46b04425aa5393515cb9f781c P 29806d3d37354c169484f6db368f8798 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } } }
I20240502 01:54:39.949651 19466 raft_consensus.cc:2932] T 0dd8944f9fd542e29072ca501df5e112 P 29806d3d37354c169484f6db368f8798 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 272b2179c1b4476090a89880df2a8baa (127.9.176.199) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } } }
I20240502 01:54:39.995792 19467 raft_consensus.cc:1203] T a5e03cf44eab41cc842342a138c04067 P 29806d3d37354c169484f6db368f8798 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 01:54:40.048878 17898 consensus_queue.cc:237] T 05b67d9912f047bfb6a2bed90b82ae15 P da3584f429104d23a1a8899b35cc2138 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } }
I20240502 01:54:40.053170 19871 raft_consensus.cc:1203] T 91e0e82526c0489a8908e250bf3c851f P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240502 01:54:40.068364 17613 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 01:54:40.083218 17833 consensus_peers.cc:485] T 05b67d9912f047bfb6a2bed90b82ae15 P da3584f429104d23a1a8899b35cc2138 -> Peer 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200:37831): Couldn't send request to peer 94931f4b51c9493cb3d4264ce99f5e76. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 05b67d9912f047bfb6a2bed90b82ae15. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:40.084849 19603 raft_consensus.cc:1261] T 05b67d9912f047bfb6a2bed90b82ae15 P 4b7c382c9edd41babdd1a1afc904a813 [term 1 FOLLOWER]: Refusing update from remote peer da3584f429104d23a1a8899b35cc2138: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:40.084993 17763 raft_consensus.cc:1261] T 05b67d9912f047bfb6a2bed90b82ae15 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Refusing update from remote peer da3584f429104d23a1a8899b35cc2138: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:40.086306 20915 consensus_queue.cc:1035] T 05b67d9912f047bfb6a2bed90b82ae15 P da3584f429104d23a1a8899b35cc2138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:40.087245 20915 consensus_queue.cc:1035] T 05b67d9912f047bfb6a2bed90b82ae15 P da3584f429104d23a1a8899b35cc2138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:40.100515 19603 raft_consensus.cc:1203] T 5f3f3e5f772b4566b69236fd87d71a38 P 4b7c382c9edd41babdd1a1afc904a813 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 01:54:40.128393 17478 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:54:40.129480 17633 consensus_queue.cc:237] T 0dd8944f9fd542e29072ca501df5e112 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } }
I20240502 01:54:40.131990 19718 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:40.132391 19738 raft_consensus.cc:1261] T 0dd8944f9fd542e29072ca501df5e112 P 272b2179c1b4476090a89880df2a8baa [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:40.132934 21243 consensus_queue.cc:1035] T 0dd8944f9fd542e29072ca501df5e112 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:40.133319 17500 raft_consensus.cc:1261] T 0dd8944f9fd542e29072ca501df5e112 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:40.133360 17764 raft_consensus.cc:1261] T 0dd8944f9fd542e29072ca501df5e112 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:40.133980 21243 consensus_queue.cc:1035] T 0dd8944f9fd542e29072ca501df5e112 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:40.134449 21243 consensus_queue.cc:1035] T 0dd8944f9fd542e29072ca501df5e112 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:40.137327 21243 raft_consensus.cc:2932] T 0dd8944f9fd542e29072ca501df5e112 P 1b50a031f2784ba498b3e52e16916205 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 29806d3d37354c169484f6db368f8798 (127.9.176.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } }
I20240502 01:54:40.138340 17500 raft_consensus.cc:2932] T 0dd8944f9fd542e29072ca501df5e112 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 29806d3d37354c169484f6db368f8798 (127.9.176.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } }
I20240502 01:54:40.138890 17764 raft_consensus.cc:2932] T 0dd8944f9fd542e29072ca501df5e112 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 29806d3d37354c169484f6db368f8798 (127.9.176.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } }
I20240502 01:54:40.144009 17343 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0dd8944f9fd542e29072ca501df5e112 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:40.147202 19736 raft_consensus.cc:2932] T 0dd8944f9fd542e29072ca501df5e112 P 272b2179c1b4476090a89880df2a8baa [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 29806d3d37354c169484f6db368f8798 (127.9.176.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } }
I20240502 01:54:40.150707 17354 catalog_manager.cc:5484] T 0dd8944f9fd542e29072ca501df5e112 P 1b50a031f2784ba498b3e52e16916205 reported cstate change: config changed from index 6 to 7, NON_VOTER 29806d3d37354c169484f6db368f8798 (127.9.176.197) evicted. New cstate: current_term: 1 leader_uuid: "1b50a031f2784ba498b3e52e16916205" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:40.160485 20915 raft_consensus.cc:2932] T 05b67d9912f047bfb6a2bed90b82ae15 P da3584f429104d23a1a8899b35cc2138 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } } }
I20240502 01:54:40.163208 17746 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:54:40.167236 17767 raft_consensus.cc:2932] T 05b67d9912f047bfb6a2bed90b82ae15 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } } }
I20240502 01:54:40.168239 19585 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:40.198643 19446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:40.203984 17354 catalog_manager.cc:5484] T 05b67d9912f047bfb6a2bed90b82ae15 P 84d3a4b174c34e02aedab1e620731359 reported cstate change: config changed from index 6 to 7, NON_VOTER 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) added. New cstate: current_term: 1 leader_uuid: "da3584f429104d23a1a8899b35cc2138" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } } }
I20240502 01:54:40.212668 17631 consensus_queue.cc:237] T 0dd8944f9fd542e29072ca501df5e112 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } }
I20240502 01:54:40.217612 19738 raft_consensus.cc:1261] T 0dd8944f9fd542e29072ca501df5e112 P 272b2179c1b4476090a89880df2a8baa [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:40.220351 21625 consensus_queue.cc:1035] T 0dd8944f9fd542e29072ca501df5e112 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:40.229491 17500 raft_consensus.cc:1261] T 0dd8944f9fd542e29072ca501df5e112 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:40.230320 21243 consensus_queue.cc:1035] T 0dd8944f9fd542e29072ca501df5e112 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.003s
I20240502 01:54:40.237843 19603 raft_consensus.cc:2932] T 05b67d9912f047bfb6a2bed90b82ae15 P 4b7c382c9edd41babdd1a1afc904a813 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } } }
I20240502 01:54:40.252151 21630 raft_consensus.cc:2932] T 0dd8944f9fd542e29072ca501df5e112 P 1b50a031f2784ba498b3e52e16916205 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 84d3a4b174c34e02aedab1e620731359 (127.9.176.195) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } }
I20240502 01:54:40.265343 17343 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0dd8944f9fd542e29072ca501df5e112 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:40.266927 19736 raft_consensus.cc:2932] T 0dd8944f9fd542e29072ca501df5e112 P 272b2179c1b4476090a89880df2a8baa [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 84d3a4b174c34e02aedab1e620731359 (127.9.176.195) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } }
I20240502 01:54:40.275395 17499 raft_consensus.cc:2932] T 0dd8944f9fd542e29072ca501df5e112 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 84d3a4b174c34e02aedab1e620731359 (127.9.176.195) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } }
I20240502 01:54:40.284502 17347 catalog_manager.cc:5484] T 0dd8944f9fd542e29072ca501df5e112 P 311f765f59754bdcb4eadc04e14bd4ea reported cstate change: config changed from index 7 to 8, VOTER 84d3a4b174c34e02aedab1e620731359 (127.9.176.195) evicted. New cstate: current_term: 1 leader_uuid: "1b50a031f2784ba498b3e52e16916205" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } }
I20240502 01:54:40.290542 17880 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:54:40.291164 17746 tablet_service.cc:1514] Processing DeleteTablet for tablet 0dd8944f9fd542e29072ca501df5e112 with delete_type TABLET_DATA_TOMBSTONED (TS 84d3a4b174c34e02aedab1e620731359 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:47166
I20240502 01:54:40.292788 21409 raft_consensus.cc:1050] T 075ded9523e44c3d92844805338643a3 P 311f765f59754bdcb4eadc04e14bd4ea: attempting to promote NON_VOTER 94931f4b51c9493cb3d4264ce99f5e76 to VOTER
I20240502 01:54:40.293741 21409 consensus_queue.cc:237] T 075ded9523e44c3d92844805338643a3 P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } }
I20240502 01:54:40.308416 21713 tablet_replica.cc:330] T 0dd8944f9fd542e29072ca501df5e112 P 84d3a4b174c34e02aedab1e620731359: stopping tablet replica
I20240502 01:54:40.308683 21713 raft_consensus.cc:2229] T 0dd8944f9fd542e29072ca501df5e112 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:40.309091 21713 raft_consensus.cc:2258] T 0dd8944f9fd542e29072ca501df5e112 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:40.316583 17631 raft_consensus.cc:1261] T 075ded9523e44c3d92844805338643a3 P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Refusing update from remote peer 311f765f59754bdcb4eadc04e14bd4ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:40.316732 17898 raft_consensus.cc:1261] T 075ded9523e44c3d92844805338643a3 P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Refusing update from remote peer 311f765f59754bdcb4eadc04e14bd4ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:40.317456 21409 consensus_queue.cc:1035] T 075ded9523e44c3d92844805338643a3 P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:40.317636 21713 ts_tablet_manager.cc:1875] T 0dd8944f9fd542e29072ca501df5e112 P 84d3a4b174c34e02aedab1e620731359: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:40.319491 21342 consensus_queue.cc:1035] T 075ded9523e44c3d92844805338643a3 P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:40.329939 19871 raft_consensus.cc:1261] T 075ded9523e44c3d92844805338643a3 P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 LEARNER]: Refusing update from remote peer 311f765f59754bdcb4eadc04e14bd4ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:40.334833 21467 consensus_queue.cc:1035] T 075ded9523e44c3d92844805338643a3 P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:40.351612 21467 raft_consensus.cc:1050] T 7ce9d4281c1b46258493feb39e8d9ab2 P 311f765f59754bdcb4eadc04e14bd4ea: attempting to promote NON_VOTER 4b7c382c9edd41babdd1a1afc904a813 to VOTER
I20240502 01:54:40.352635 21467 consensus_queue.cc:237] T 7ce9d4281c1b46258493feb39e8d9ab2 P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } }
I20240502 01:54:40.356078 21342 raft_consensus.cc:2932] T 075ded9523e44c3d92844805338643a3 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
I20240502 01:54:40.361202 19871 raft_consensus.cc:2932] T 075ded9523e44c3d92844805338643a3 P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
I20240502 01:54:40.368851 17898 raft_consensus.cc:2932] T 075ded9523e44c3d92844805338643a3 P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
I20240502 01:54:40.380137 19446 tablet_service.cc:1514] Processing DeleteTablet for tablet 0dd8944f9fd542e29072ca501df5e112 with delete_type TABLET_DATA_TOMBSTONED (TS 29806d3d37354c169484f6db368f8798 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:33166
I20240502 01:54:40.384706 19603 raft_consensus.cc:1261] T 7ce9d4281c1b46258493feb39e8d9ab2 P 4b7c382c9edd41babdd1a1afc904a813 [term 1 LEARNER]: Refusing update from remote peer 311f765f59754bdcb4eadc04e14bd4ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:40.385548 17351 catalog_manager.cc:5484] T 075ded9523e44c3d92844805338643a3 P da3584f429104d23a1a8899b35cc2138 reported cstate change: config changed from index 4 to 5, 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "311f765f59754bdcb4eadc04e14bd4ea" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
I20240502 01:54:40.386284 17633 raft_consensus.cc:1261] T 7ce9d4281c1b46258493feb39e8d9ab2 P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Refusing update from remote peer 311f765f59754bdcb4eadc04e14bd4ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:40.386253 21409 consensus_queue.cc:1035] T 7ce9d4281c1b46258493feb39e8d9ab2 P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:40.387815 21732 consensus_queue.cc:1035] T 7ce9d4281c1b46258493feb39e8d9ab2 P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240502 01:54:40.391191 17633 consensus_queue.cc:237] T 3b3fa060e8d64911bec810ec07a358d2 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { promote: true } }
I20240502 01:54:40.393100 21733 tablet_replica.cc:330] T 0dd8944f9fd542e29072ca501df5e112 P 29806d3d37354c169484f6db368f8798: stopping tablet replica
I20240502 01:54:40.394563 21733 raft_consensus.cc:2229] T 0dd8944f9fd542e29072ca501df5e112 P 29806d3d37354c169484f6db368f8798 [term 1 LEARNER]: Raft consensus shutting down.
I20240502 01:54:40.397833 21713 ts_tablet_manager.cc:1888] T 0dd8944f9fd542e29072ca501df5e112 P 84d3a4b174c34e02aedab1e620731359: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240502 01:54:40.397817 21733 raft_consensus.cc:2258] T 0dd8944f9fd542e29072ca501df5e112 P 29806d3d37354c169484f6db368f8798 [term 1 LEARNER]: Raft consensus is shut down!
I20240502 01:54:40.399356 21707 raft_consensus.cc:1050] T a5e03cf44eab41cc842342a138c04067 P da3584f429104d23a1a8899b35cc2138: attempting to promote NON_VOTER 29806d3d37354c169484f6db368f8798 to VOTER
I20240502 01:54:40.400645 21707 consensus_queue.cc:237] T a5e03cf44eab41cc842342a138c04067 P da3584f429104d23a1a8899b35cc2138 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } }
I20240502 01:54:40.403172 21713 log.cc:1196] T 0dd8944f9fd542e29072ca501df5e112 P 84d3a4b174c34e02aedab1e620731359: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/wal/wals/0dd8944f9fd542e29072ca501df5e112
I20240502 01:54:40.408238 17898 raft_consensus.cc:1261] T 7ce9d4281c1b46258493feb39e8d9ab2 P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Refusing update from remote peer 311f765f59754bdcb4eadc04e14bd4ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:40.408288 17634 consensus_queue.cc:237] T 2c78951695284b1aaab411c0a041c28a P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } }
I20240502 01:54:40.409075 21409 consensus_queue.cc:1035] T 7ce9d4281c1b46258493feb39e8d9ab2 P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:40.413336 17343 catalog_manager.cc:4830] TS 84d3a4b174c34e02aedab1e620731359 (127.9.176.195:37357): tablet 0dd8944f9fd542e29072ca501df5e112 (table rebalance_test_table_9 [id=60aa7e7a81564c3a888cc9b68700c9ad]) successfully deleted
I20240502 01:54:40.417579 19852 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:54:40.437822 17565 consensus_peers.cc:485] T 3b3fa060e8d64911bec810ec07a358d2 P 1b50a031f2784ba498b3e52e16916205 -> Peer da3584f429104d23a1a8899b35cc2138 (127.9.176.196:46381): Couldn't send request to peer da3584f429104d23a1a8899b35cc2138. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:40.439038 17631 raft_consensus.cc:2932] T 075ded9523e44c3d92844805338643a3 P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
I20240502 01:54:40.459524 21409 raft_consensus.cc:2932] T 7ce9d4281c1b46258493feb39e8d9ab2 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
I20240502 01:54:40.462795 17901 raft_consensus.cc:2932] T 7ce9d4281c1b46258493feb39e8d9ab2 P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
I20240502 01:54:40.464974 19871 raft_consensus.cc:1261] T 2c78951695284b1aaab411c0a041c28a P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:54:40.468731 21741 ts_tablet_manager.cc:902] T c91f74f46b04425aa5393515cb9f781c P 4b7c382c9edd41babdd1a1afc904a813: Initiating tablet copy from peer 84d3a4b174c34e02aedab1e620731359 (127.9.176.195:37357)
I20240502 01:54:40.473605 17347 catalog_manager.cc:5484] T 7ce9d4281c1b46258493feb39e8d9ab2 P da3584f429104d23a1a8899b35cc2138 reported cstate change: config changed from index 4 to 5, 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "311f765f59754bdcb4eadc04e14bd4ea" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
I20240502 01:54:40.481434 17764 consensus_queue.cc:237] T 6da59c0626c64a5988e39a831c42c268 P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } }
I20240502 01:54:40.486016 21733 ts_tablet_manager.cc:1875] T 0dd8944f9fd542e29072ca501df5e112 P 29806d3d37354c169484f6db368f8798: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:40.509371 21741 tablet_copy_client.cc:308] T c91f74f46b04425aa5393515cb9f781c P 4b7c382c9edd41babdd1a1afc904a813: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.195:37357
I20240502 01:54:40.510331 17789 tablet_copy_service.cc:140] P 84d3a4b174c34e02aedab1e620731359: Received BeginTabletCopySession request for tablet c91f74f46b04425aa5393515cb9f781c from peer 4b7c382c9edd41babdd1a1afc904a813 ({username='slave'} at 127.9.176.198:55177)
I20240502 01:54:40.510578 17789 tablet_copy_service.cc:161] P 84d3a4b174c34e02aedab1e620731359: Beginning new tablet copy session on tablet c91f74f46b04425aa5393515cb9f781c from peer 4b7c382c9edd41babdd1a1afc904a813 at {username='slave'} at 127.9.176.198:55177: session id = 4b7c382c9edd41babdd1a1afc904a813-c91f74f46b04425aa5393515cb9f781c
I20240502 01:54:40.513748 17789 tablet_copy_source_session.cc:205] T c91f74f46b04425aa5393515cb9f781c P 84d3a4b174c34e02aedab1e620731359: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:40.515398 21741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c91f74f46b04425aa5393515cb9f781c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:40.521090 17500 consensus_queue.cc:237] T 075ded9523e44c3d92844805338643a3 P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } }
I20240502 01:54:40.467814 21708 consensus_queue.cc:1035] T 2c78951695284b1aaab411c0a041c28a P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.002s
I20240502 01:54:40.526078 17499 raft_consensus.cc:1261] T 2c78951695284b1aaab411c0a041c28a P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:54:40.528007 20628 consensus_queue.cc:1035] T 2c78951695284b1aaab411c0a041c28a P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:40.528959 20628 raft_consensus.cc:1050] T 91e0e82526c0489a8908e250bf3c851f P 1b50a031f2784ba498b3e52e16916205: attempting to promote NON_VOTER 94931f4b51c9493cb3d4264ce99f5e76 to VOTER
I20240502 01:54:40.529894 20628 consensus_queue.cc:237] T 91e0e82526c0489a8908e250bf3c851f P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } }
I20240502 01:54:40.533754 21733 ts_tablet_manager.cc:1888] T 0dd8944f9fd542e29072ca501df5e112 P 29806d3d37354c169484f6db368f8798: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 01:54:40.533980 21733 log.cc:1196] T 0dd8944f9fd542e29072ca501df5e112 P 29806d3d37354c169484f6db368f8798: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/wal/wals/0dd8944f9fd542e29072ca501df5e112
I20240502 01:54:40.535162 17340 catalog_manager.cc:4830] TS 29806d3d37354c169484f6db368f8798 (127.9.176.197:35011): tablet 0dd8944f9fd542e29072ca501df5e112 (table rebalance_test_table_9 [id=60aa7e7a81564c3a888cc9b68700c9ad]) successfully deleted
I20240502 01:54:40.535890 19736 raft_consensus.cc:1261] T 2c78951695284b1aaab411c0a041c28a P 272b2179c1b4476090a89880df2a8baa [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:54:40.536993 21625 consensus_queue.cc:1035] T 2c78951695284b1aaab411c0a041c28a P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:40.550230 19872 raft_consensus.cc:1261] T 075ded9523e44c3d92844805338643a3 P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Refusing update from remote peer 311f765f59754bdcb4eadc04e14bd4ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:40.556118 21342 consensus_queue.cc:1035] T 075ded9523e44c3d92844805338643a3 P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:40.561178 19872 raft_consensus.cc:2932] T 075ded9523e44c3d92844805338643a3 P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER da3584f429104d23a1a8899b35cc2138 (127.9.176.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
I20240502 01:54:40.571127 21741 tablet_copy_client.cc:783] T c91f74f46b04425aa5393515cb9f781c P 4b7c382c9edd41babdd1a1afc904a813: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:40.572782 21467 raft_consensus.cc:2932] T 075ded9523e44c3d92844805338643a3 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER da3584f429104d23a1a8899b35cc2138 (127.9.176.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
I20240502 01:54:40.577823 19467 raft_consensus.cc:1261] T a5e03cf44eab41cc842342a138c04067 P 29806d3d37354c169484f6db368f8798 [term 1 LEARNER]: Refusing update from remote peer da3584f429104d23a1a8899b35cc2138: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:40.578939 21695 consensus_queue.cc:1035] T a5e03cf44eab41cc842342a138c04067 P da3584f429104d23a1a8899b35cc2138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:40.583077 17500 raft_consensus.cc:1261] T 91e0e82526c0489a8908e250bf3c851f P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:40.583441 19465 raft_consensus.cc:1261] T 91e0e82526c0489a8908e250bf3c851f P 29806d3d37354c169484f6db368f8798 [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:40.584443 21625 consensus_queue.cc:1035] T 91e0e82526c0489a8908e250bf3c851f P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:40.585637 21625 consensus_queue.cc:1035] T 91e0e82526c0489a8908e250bf3c851f P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:40.592401 17498 raft_consensus.cc:1261] T a5e03cf44eab41cc842342a138c04067 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Refusing update from remote peer da3584f429104d23a1a8899b35cc2138: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:40.593418 21695 consensus_queue.cc:1035] T a5e03cf44eab41cc842342a138c04067 P da3584f429104d23a1a8899b35cc2138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:40.597826 19872 raft_consensus.cc:1261] T 91e0e82526c0489a8908e250bf3c851f P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 LEARNER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:40.600767 21625 consensus_queue.cc:1035] T 91e0e82526c0489a8908e250bf3c851f P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:40.606720 17763 raft_consensus.cc:1261] T a5e03cf44eab41cc842342a138c04067 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Refusing update from remote peer da3584f429104d23a1a8899b35cc2138: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:40.607473 21695 consensus_queue.cc:1035] T a5e03cf44eab41cc842342a138c04067 P da3584f429104d23a1a8899b35cc2138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:40.621109 21741 tablet_copy_client.cc:647] T c91f74f46b04425aa5393515cb9f781c P 4b7c382c9edd41babdd1a1afc904a813: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:40.631441 21630 raft_consensus.cc:2932] T 2c78951695284b1aaab411c0a041c28a P 1b50a031f2784ba498b3e52e16916205 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
I20240502 01:54:40.632525 19736 raft_consensus.cc:2932] T 2c78951695284b1aaab411c0a041c28a P 272b2179c1b4476090a89880df2a8baa [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
I20240502 01:54:40.635504 17497 raft_consensus.cc:2932] T 2c78951695284b1aaab411c0a041c28a P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
I20240502 01:54:40.644548 19871 raft_consensus.cc:2932] T 2c78951695284b1aaab411c0a041c28a P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
I20240502 01:54:40.645233 17347 catalog_manager.cc:5484] T 2c78951695284b1aaab411c0a041c28a P 1b50a031f2784ba498b3e52e16916205 reported cstate change: config changed from index 8 to 9. New cstate: current_term: 1 leader_uuid: "1b50a031f2784ba498b3e52e16916205" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:40.650672 21243 raft_consensus.cc:2932] T 91e0e82526c0489a8908e250bf3c851f P 1b50a031f2784ba498b3e52e16916205 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
I20240502 01:54:40.655289 17632 consensus_queue.cc:237] T 2c78951695284b1aaab411c0a041c28a P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } }
I20240502 01:54:40.656780 19872 raft_consensus.cc:2932] T 91e0e82526c0489a8908e250bf3c851f P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
I20240502 01:54:40.658942 19871 raft_consensus.cc:1261] T 2c78951695284b1aaab411c0a041c28a P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:54:40.659637 21708 consensus_queue.cc:1035] T 2c78951695284b1aaab411c0a041c28a P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:54:40.662814 21741 tablet_copy_client.cc:523] T c91f74f46b04425aa5393515cb9f781c P 4b7c382c9edd41babdd1a1afc904a813: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:40.662945 21708 raft_consensus.cc:2932] T 2c78951695284b1aaab411c0a041c28a P 1b50a031f2784ba498b3e52e16916205 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
I20240502 01:54:40.668334 19869 raft_consensus.cc:2932] T 2c78951695284b1aaab411c0a041c28a P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
I20240502 01:54:40.669011 21741 tablet_bootstrap.cc:492] T c91f74f46b04425aa5393515cb9f781c P 4b7c382c9edd41babdd1a1afc904a813: Bootstrap starting.
I20240502 01:54:40.672757 17343 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2c78951695284b1aaab411c0a041c28a with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:40.675364 21625 raft_consensus.cc:1050] T 5f3f3e5f772b4566b69236fd87d71a38 P 1b50a031f2784ba498b3e52e16916205: attempting to promote NON_VOTER 4b7c382c9edd41babdd1a1afc904a813 to VOTER
I20240502 01:54:40.676422 21625 consensus_queue.cc:237] T 5f3f3e5f772b4566b69236fd87d71a38 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } }
I20240502 01:54:40.677219 17500 raft_consensus.cc:2932] T 91e0e82526c0489a8908e250bf3c851f P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
I20240502 01:54:40.681360 17347 catalog_manager.cc:5484] T 2c78951695284b1aaab411c0a041c28a P 1b50a031f2784ba498b3e52e16916205 reported cstate change: config changed from index 9 to 10, VOTER 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193) evicted. New cstate: current_term: 1 leader_uuid: "1b50a031f2784ba498b3e52e16916205" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:40.682324 17347 catalog_manager.cc:5484] T 91e0e82526c0489a8908e250bf3c851f P 1b50a031f2784ba498b3e52e16916205 reported cstate change: config changed from index 7 to 8, 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1b50a031f2784ba498b3e52e16916205" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:40.687505 19604 raft_consensus.cc:2932] T 7ce9d4281c1b46258493feb39e8d9ab2 P 4b7c382c9edd41babdd1a1afc904a813 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
I20240502 01:54:40.691582 19738 raft_consensus.cc:1261] T 2c78951695284b1aaab411c0a041c28a P 272b2179c1b4476090a89880df2a8baa [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:54:40.695035 20628 consensus_queue.cc:1035] T 2c78951695284b1aaab411c0a041c28a P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:54:40.697548 19738 raft_consensus.cc:2932] T 2c78951695284b1aaab411c0a041c28a P 272b2179c1b4476090a89880df2a8baa [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
I20240502 01:54:40.701239 17763 raft_consensus.cc:1261] T 5f3f3e5f772b4566b69236fd87d71a38 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:40.701864 21630 consensus_queue.cc:1035] T 5f3f3e5f772b4566b69236fd87d71a38 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:40.704907 17478 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c78951695284b1aaab411c0a041c28a with delete_type TABLET_DATA_TOMBSTONED (TS 311f765f59754bdcb4eadc04e14bd4ea not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:48974
I20240502 01:54:40.705233 17898 raft_consensus.cc:1261] T 5f3f3e5f772b4566b69236fd87d71a38 P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:40.706012 21625 consensus_queue.cc:1035] T 5f3f3e5f772b4566b69236fd87d71a38 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:40.710029 17632 consensus_queue.cc:237] T 2c78951695284b1aaab411c0a041c28a P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } }
I20240502 01:54:40.712471 19603 raft_consensus.cc:1261] T 5f3f3e5f772b4566b69236fd87d71a38 P 4b7c382c9edd41babdd1a1afc904a813 [term 1 LEARNER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:40.713471 21630 consensus_queue.cc:1035] T 5f3f3e5f772b4566b69236fd87d71a38 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:40.720852 19737 raft_consensus.cc:1261] T 2c78951695284b1aaab411c0a041c28a P 272b2179c1b4476090a89880df2a8baa [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
W20240502 01:54:40.721159 17565 consensus_peers.cc:485] T 2c78951695284b1aaab411c0a041c28a P 1b50a031f2784ba498b3e52e16916205 -> Peer 29806d3d37354c169484f6db368f8798 (127.9.176.197:35011): Couldn't send request to peer 29806d3d37354c169484f6db368f8798. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2c78951695284b1aaab411c0a041c28a. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:40.721771 21243 consensus_queue.cc:1035] T 2c78951695284b1aaab411c0a041c28a P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 01:54:40.725083 21708 raft_consensus.cc:2932] T 2c78951695284b1aaab411c0a041c28a P 1b50a031f2784ba498b3e52e16916205 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 29806d3d37354c169484f6db368f8798 (127.9.176.197) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } } }
I20240502 01:54:40.725719 19737 raft_consensus.cc:2932] T 2c78951695284b1aaab411c0a041c28a P 272b2179c1b4476090a89880df2a8baa [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 29806d3d37354c169484f6db368f8798 (127.9.176.197) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } } }
I20240502 01:54:40.726886 19869 raft_consensus.cc:1261] T 2c78951695284b1aaab411c0a041c28a P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 01:54:40.727681 21630 consensus_queue.cc:1035] T 2c78951695284b1aaab411c0a041c28a P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 01:54:40.729972 17343 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2c78951695284b1aaab411c0a041c28a with cas_config_opid_index 10: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 01:54:40.732622 19870 raft_consensus.cc:2932] T 2c78951695284b1aaab411c0a041c28a P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 29806d3d37354c169484f6db368f8798 (127.9.176.197) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } } }
I20240502 01:54:40.734656 17348 catalog_manager.cc:5484] T 2c78951695284b1aaab411c0a041c28a P 272b2179c1b4476090a89880df2a8baa reported cstate change: config changed from index 10 to 11, NON_VOTER 29806d3d37354c169484f6db368f8798 (127.9.176.197) added. New cstate: current_term: 1 leader_uuid: "1b50a031f2784ba498b3e52e16916205" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } } }
I20240502 01:54:40.751169 21771 ts_tablet_manager.cc:902] T 05b67d9912f047bfb6a2bed90b82ae15 P 94931f4b51c9493cb3d4264ce99f5e76: Initiating tablet copy from peer da3584f429104d23a1a8899b35cc2138 (127.9.176.196:46381)
I20240502 01:54:40.752755 21243 raft_consensus.cc:2932] T 5f3f3e5f772b4566b69236fd87d71a38 P 1b50a031f2784ba498b3e52e16916205 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
I20240502 01:54:40.758466 17632 raft_consensus.cc:2932] T 7ce9d4281c1b46258493feb39e8d9ab2 P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
I20240502 01:54:40.758375 19603 raft_consensus.cc:2932] T 5f3f3e5f772b4566b69236fd87d71a38 P 4b7c382c9edd41babdd1a1afc904a813 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
I20240502 01:54:40.761642 17351 catalog_manager.cc:5484] T 075ded9523e44c3d92844805338643a3 P 94931f4b51c9493cb3d4264ce99f5e76 reported cstate change: config changed from index 5 to 6, VOTER da3584f429104d23a1a8899b35cc2138 (127.9.176.196) evicted. New cstate: current_term: 1 leader_uuid: "311f765f59754bdcb4eadc04e14bd4ea" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
I20240502 01:54:40.771742 17349 catalog_manager.cc:5484] T 5f3f3e5f772b4566b69236fd87d71a38 P 1b50a031f2784ba498b3e52e16916205 reported cstate change: config changed from index 4 to 5, 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1b50a031f2784ba498b3e52e16916205" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:40.773365 19872 raft_consensus.cc:1261] T 6da59c0626c64a5988e39a831c42c268 P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:54:40.774155 21362 consensus_queue.cc:1035] T 6da59c0626c64a5988e39a831c42c268 P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:54:40.775209 17901 raft_consensus.cc:2932] T 5f3f3e5f772b4566b69236fd87d71a38 P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
I20240502 01:54:40.777508 17632 consensus_queue.cc:237] T 5f3f3e5f772b4566b69236fd87d71a38 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } }
I20240502 01:54:40.781577 19466 raft_consensus.cc:1261] T 6da59c0626c64a5988e39a831c42c268 P 29806d3d37354c169484f6db368f8798 [term 1 FOLLOWER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
W20240502 01:54:40.782963 17700 consensus_peers.cc:485] T 6da59c0626c64a5988e39a831c42c268 P 84d3a4b174c34e02aedab1e620731359 -> Peer 272b2179c1b4476090a89880df2a8baa (127.9.176.199:42269): Couldn't send request to peer 272b2179c1b4476090a89880df2a8baa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6da59c0626c64a5988e39a831c42c268. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:40.783591 21362 consensus_queue.cc:1035] T 6da59c0626c64a5988e39a831c42c268 P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20240502 01:54:40.790731 17499 raft_consensus.cc:1261] T 3b3fa060e8d64911bec810ec07a358d2 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:40.793802 19736 raft_consensus.cc:1261] T 3b3fa060e8d64911bec810ec07a358d2 P 272b2179c1b4476090a89880df2a8baa [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:40.797713 17898 raft_consensus.cc:1261] T 5f3f3e5f772b4566b69236fd87d71a38 P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:40.798254 21760 raft_consensus.cc:2932] T 6da59c0626c64a5988e39a831c42c268 P 84d3a4b174c34e02aedab1e620731359 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 272b2179c1b4476090a89880df2a8baa (127.9.176.199) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } } }
I20240502 01:54:40.804389 19466 raft_consensus.cc:2932] T 6da59c0626c64a5988e39a831c42c268 P 29806d3d37354c169484f6db368f8798 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 272b2179c1b4476090a89880df2a8baa (127.9.176.199) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } } }
I20240502 01:54:40.807814 19604 raft_consensus.cc:1261] T 5f3f3e5f772b4566b69236fd87d71a38 P 4b7c382c9edd41babdd1a1afc904a813 [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:40.811797 20915 raft_consensus.cc:2932] T a5e03cf44eab41cc842342a138c04067 P da3584f429104d23a1a8899b35cc2138 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 29806d3d37354c169484f6db368f8798 (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } }
I20240502 01:54:40.814539 17343 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 075ded9523e44c3d92844805338643a3 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:40.816321 17764 consensus_queue.cc:237] T f07e9b433e554c85b3c25b7e708f8b2f P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } }
I20240502 01:54:40.816874 21766 tablet_replica.cc:330] T 2c78951695284b1aaab411c0a041c28a P 311f765f59754bdcb4eadc04e14bd4ea: stopping tablet replica
I20240502 01:54:40.817121 21766 raft_consensus.cc:2229] T 2c78951695284b1aaab411c0a041c28a P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:40.817646 21766 raft_consensus.cc:2258] T 2c78951695284b1aaab411c0a041c28a P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:40.823756 17498 raft_consensus.cc:2932] T a5e03cf44eab41cc842342a138c04067 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 29806d3d37354c169484f6db368f8798 (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } }
I20240502 01:54:40.826788 19467 raft_consensus.cc:2932] T a5e03cf44eab41cc842342a138c04067 P 29806d3d37354c169484f6db368f8798 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 29806d3d37354c169484f6db368f8798 (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } }
I20240502 01:54:40.829852 21630 consensus_queue.cc:1035] T 3b3fa060e8d64911bec810ec07a358d2 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:40.832860 17634 raft_consensus.cc:1261] T 075ded9523e44c3d92844805338643a3 P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Refusing update from remote peer 311f765f59754bdcb4eadc04e14bd4ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:40.833305 21708 consensus_queue.cc:1035] T 5f3f3e5f772b4566b69236fd87d71a38 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:40.833786 21708 consensus_queue.cc:1035] T 5f3f3e5f772b4566b69236fd87d71a38 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:40.834215 21708 consensus_queue.cc:1035] T 3b3fa060e8d64911bec810ec07a358d2 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:40.835803 19870 raft_consensus.cc:2932] T 6da59c0626c64a5988e39a831c42c268 P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 272b2179c1b4476090a89880df2a8baa (127.9.176.199) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } } }
I20240502 01:54:40.839223 21625 raft_consensus.cc:2932] T 5f3f3e5f772b4566b69236fd87d71a38 P 1b50a031f2784ba498b3e52e16916205 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 84d3a4b174c34e02aedab1e620731359 (127.9.176.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
I20240502 01:54:40.842979 17767 raft_consensus.cc:2932] T a5e03cf44eab41cc842342a138c04067 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 29806d3d37354c169484f6db368f8798 (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } }
I20240502 01:54:40.843166 17347 catalog_manager.cc:5484] T a5e03cf44eab41cc842342a138c04067 P da3584f429104d23a1a8899b35cc2138 reported cstate change: config changed from index 4 to 5, 29806d3d37354c169484f6db368f8798 (127.9.176.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "da3584f429104d23a1a8899b35cc2138" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:40.846182 17343 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5f3f3e5f772b4566b69236fd87d71a38 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:40.850862 19603 raft_consensus.cc:2932] T 5f3f3e5f772b4566b69236fd87d71a38 P 4b7c382c9edd41babdd1a1afc904a813 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 84d3a4b174c34e02aedab1e620731359 (127.9.176.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
I20240502 01:54:40.855650 21766 ts_tablet_manager.cc:1875] T 2c78951695284b1aaab411c0a041c28a P 311f765f59754bdcb4eadc04e14bd4ea: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:40.860034 17348 catalog_manager.cc:5484] T 6da59c0626c64a5988e39a831c42c268 P 84d3a4b174c34e02aedab1e620731359 reported cstate change: config changed from index 9 to 10, NON_VOTER 272b2179c1b4476090a89880df2a8baa (127.9.176.199) added. New cstate: current_term: 1 leader_uuid: "84d3a4b174c34e02aedab1e620731359" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:40.860980 17880 tablet_service.cc:1514] Processing DeleteTablet for tablet 075ded9523e44c3d92844805338643a3 with delete_type TABLET_DATA_TOMBSTONED (TS da3584f429104d23a1a8899b35cc2138 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:34396
W20240502 01:54:40.862587 17700 consensus_peers.cc:485] T f07e9b433e554c85b3c25b7e708f8b2f P 84d3a4b174c34e02aedab1e620731359 -> Peer 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198:36865): Couldn't send request to peer 4b7c382c9edd41babdd1a1afc904a813. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f07e9b433e554c85b3c25b7e708f8b2f. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:40.855156 17901 raft_consensus.cc:2932] T 5f3f3e5f772b4566b69236fd87d71a38 P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 84d3a4b174c34e02aedab1e620731359 (127.9.176.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
I20240502 01:54:40.870214 19872 raft_consensus.cc:1261] T f07e9b433e554c85b3c25b7e708f8b2f P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:40.871212 21362 consensus_queue.cc:1035] T f07e9b433e554c85b3c25b7e708f8b2f P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:40.874800 21734 consensus_queue.cc:1035] T 075ded9523e44c3d92844805338643a3 P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:40.876576 17631 raft_consensus.cc:1261] T f07e9b433e554c85b3c25b7e708f8b2f P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:40.877226 17634 raft_consensus.cc:2932] T 075ded9523e44c3d92844805338643a3 P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER da3584f429104d23a1a8899b35cc2138 (127.9.176.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
I20240502 01:54:40.882588 21362 consensus_queue.cc:1035] T f07e9b433e554c85b3c25b7e708f8b2f P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:40.892096 19465 raft_consensus.cc:2932] T 91e0e82526c0489a8908e250bf3c851f P 29806d3d37354c169484f6db368f8798 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
I20240502 01:54:40.897078 17898 consensus_queue.cc:237] T a5e03cf44eab41cc842342a138c04067 P da3584f429104d23a1a8899b35cc2138 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } }
I20240502 01:54:40.905463 21787 tablet_replica.cc:330] T 075ded9523e44c3d92844805338643a3 P da3584f429104d23a1a8899b35cc2138: stopping tablet replica
I20240502 01:54:40.906406 21787 raft_consensus.cc:2229] T 075ded9523e44c3d92844805338643a3 P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:40.906744 21787 raft_consensus.cc:2258] T 075ded9523e44c3d92844805338643a3 P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:40.908524 17764 raft_consensus.cc:1261] T a5e03cf44eab41cc842342a138c04067 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Refusing update from remote peer da3584f429104d23a1a8899b35cc2138: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:40.914654 21777 raft_consensus.cc:2932] T f07e9b433e554c85b3c25b7e708f8b2f P 84d3a4b174c34e02aedab1e620731359 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } } }
I20240502 01:54:40.916177 17634 raft_consensus.cc:2932] T f07e9b433e554c85b3c25b7e708f8b2f P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } } }
I20240502 01:54:40.917811 21708 raft_consensus.cc:2932] T 3b3fa060e8d64911bec810ec07a358d2 P 1b50a031f2784ba498b3e52e16916205 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER da3584f429104d23a1a8899b35cc2138 (127.9.176.196) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { promote: true } } }
I20240502 01:54:40.920296 19869 raft_consensus.cc:2932] T f07e9b433e554c85b3c25b7e708f8b2f P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } } }
I20240502 01:54:40.925961 17349 catalog_manager.cc:5484] T 5f3f3e5f772b4566b69236fd87d71a38 P da3584f429104d23a1a8899b35cc2138 reported cstate change: config changed from index 5 to 6, VOTER 84d3a4b174c34e02aedab1e620731359 (127.9.176.195) evicted. New cstate: current_term: 1 leader_uuid: "1b50a031f2784ba498b3e52e16916205" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
I20240502 01:54:40.929525 17500 raft_consensus.cc:2932] T 3b3fa060e8d64911bec810ec07a358d2 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER da3584f429104d23a1a8899b35cc2138 (127.9.176.196) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { promote: true } } }
I20240502 01:54:40.938179 19738 raft_consensus.cc:2932] T 3b3fa060e8d64911bec810ec07a358d2 P 272b2179c1b4476090a89880df2a8baa [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER da3584f429104d23a1a8899b35cc2138 (127.9.176.196) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { promote: true } } }
I20240502 01:54:40.909639 21778 consensus_queue.cc:1035] T a5e03cf44eab41cc842342a138c04067 P da3584f429104d23a1a8899b35cc2138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:40.941982 19466 raft_consensus.cc:1261] T a5e03cf44eab41cc842342a138c04067 P 29806d3d37354c169484f6db368f8798 [term 1 FOLLOWER]: Refusing update from remote peer da3584f429104d23a1a8899b35cc2138: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:40.945130 17348 catalog_manager.cc:5484] T f07e9b433e554c85b3c25b7e708f8b2f P 94931f4b51c9493cb3d4264ce99f5e76 reported cstate change: config changed from index 6 to 7, NON_VOTER 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) added. New cstate: current_term: 1 leader_uuid: "84d3a4b174c34e02aedab1e620731359" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } } }
I20240502 01:54:40.952142 17746 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f3f3e5f772b4566b69236fd87d71a38 with delete_type TABLET_DATA_TOMBSTONED (TS 84d3a4b174c34e02aedab1e620731359 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47166
I20240502 01:54:40.952212 21707 consensus_queue.cc:1035] T a5e03cf44eab41cc842342a138c04067 P da3584f429104d23a1a8899b35cc2138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240502 01:54:40.948565 17347 catalog_manager.cc:5484] T 3b3fa060e8d64911bec810ec07a358d2 P 272b2179c1b4476090a89880df2a8baa reported cstate change: config changed from index 6 to 7, NON_VOTER da3584f429104d23a1a8899b35cc2138 (127.9.176.196) added. New cstate: current_term: 1 leader_uuid: "1b50a031f2784ba498b3e52e16916205" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { promote: true } } }
I20240502 01:54:40.966869 17634 consensus_queue.cc:237] T 91e0e82526c0489a8908e250bf3c851f P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } }
I20240502 01:54:40.968636 21707 raft_consensus.cc:2932] T a5e03cf44eab41cc842342a138c04067 P da3584f429104d23a1a8899b35cc2138 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } }
I20240502 01:54:40.971518 17764 raft_consensus.cc:2932] T a5e03cf44eab41cc842342a138c04067 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } }
I20240502 01:54:40.972644 21766 ts_tablet_manager.cc:1888] T 2c78951695284b1aaab411c0a041c28a P 311f765f59754bdcb4eadc04e14bd4ea: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240502 01:54:40.972819 21766 log.cc:1196] T 2c78951695284b1aaab411c0a041c28a P 311f765f59754bdcb4eadc04e14bd4ea: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/wal/wals/2c78951695284b1aaab411c0a041c28a
I20240502 01:54:40.973791 17343 catalog_manager.cc:4830] TS 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193:35397): tablet 2c78951695284b1aaab411c0a041c28a (table rebalance_test_table_7 [id=1bd69df0513c4c4db4dd6d6526efee4b]) successfully deleted
I20240502 01:54:40.977217 19465 raft_consensus.cc:1261] T 91e0e82526c0489a8908e250bf3c851f P 29806d3d37354c169484f6db368f8798 [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:54:40.977484 19870 raft_consensus.cc:1261] T 91e0e82526c0489a8908e250bf3c851f P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:54:40.979648 21243 consensus_queue.cc:1035] T 91e0e82526c0489a8908e250bf3c851f P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:40.985966 21708 consensus_queue.cc:1035] T 91e0e82526c0489a8908e250bf3c851f P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:40.985929 19466 raft_consensus.cc:2932] T a5e03cf44eab41cc842342a138c04067 P 29806d3d37354c169484f6db368f8798 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } }
I20240502 01:54:40.994130 21797 tablet_replica.cc:330] T 5f3f3e5f772b4566b69236fd87d71a38 P 84d3a4b174c34e02aedab1e620731359: stopping tablet replica
I20240502 01:54:40.993921 21625 raft_consensus.cc:2932] T 91e0e82526c0489a8908e250bf3c851f P 1b50a031f2784ba498b3e52e16916205 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
I20240502 01:54:40.994616 21797 raft_consensus.cc:2229] T 5f3f3e5f772b4566b69236fd87d71a38 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:40.994961 21797 pending_rounds.cc:62] T 5f3f3e5f772b4566b69236fd87d71a38 P 84d3a4b174c34e02aedab1e620731359: Trying to abort 1 pending ops.
I20240502 01:54:40.994849 19466 raft_consensus.cc:2932] T 91e0e82526c0489a8908e250bf3c851f P 29806d3d37354c169484f6db368f8798 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
I20240502 01:54:40.995126 21797 pending_rounds.cc:68] T 5f3f3e5f772b4566b69236fd87d71a38 P 84d3a4b174c34e02aedab1e620731359: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7023062551247867904 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "5f3f3e5f772b4566b69236fd87d71a38" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } } }
I20240502 01:54:40.998195 21797 raft_consensus.cc:2894] T 5f3f3e5f772b4566b69236fd87d71a38 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240502 01:54:40.998452 21797 raft_consensus.cc:2866] T 5f3f3e5f772b4566b69236fd87d71a38 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240502 01:54:40.998467 17343 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 91e0e82526c0489a8908e250bf3c851f with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:40.998704 21797 raft_consensus.cc:2258] T 5f3f3e5f772b4566b69236fd87d71a38 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:40.999977 21797 ts_tablet_manager.cc:1875] T 5f3f3e5f772b4566b69236fd87d71a38 P 84d3a4b174c34e02aedab1e620731359: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:41.000233 21771 tablet_copy_client.cc:308] T 05b67d9912f047bfb6a2bed90b82ae15 P 94931f4b51c9493cb3d4264ce99f5e76: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.196:46381
I20240502 01:54:41.001583 17922 tablet_copy_service.cc:140] P da3584f429104d23a1a8899b35cc2138: Received BeginTabletCopySession request for tablet 05b67d9912f047bfb6a2bed90b82ae15 from peer 94931f4b51c9493cb3d4264ce99f5e76 ({username='slave'} at 127.9.176.200:36935)
I20240502 01:54:41.001785 17922 tablet_copy_service.cc:161] P da3584f429104d23a1a8899b35cc2138: Beginning new tablet copy session on tablet 05b67d9912f047bfb6a2bed90b82ae15 from peer 94931f4b51c9493cb3d4264ce99f5e76 at {username='slave'} at 127.9.176.200:36935: session id = 94931f4b51c9493cb3d4264ce99f5e76-05b67d9912f047bfb6a2bed90b82ae15
I20240502 01:54:41.001467 17351 catalog_manager.cc:5484] T 91e0e82526c0489a8908e250bf3c851f P 1b50a031f2784ba498b3e52e16916205 reported cstate change: config changed from index 8 to 9, VOTER 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193) evicted. New cstate: current_term: 1 leader_uuid: "1b50a031f2784ba498b3e52e16916205" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:41.005708 21787 ts_tablet_manager.cc:1875] T 075ded9523e44c3d92844805338643a3 P da3584f429104d23a1a8899b35cc2138: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:41.008145 17922 tablet_copy_source_session.cc:205] T 05b67d9912f047bfb6a2bed90b82ae15 P da3584f429104d23a1a8899b35cc2138: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:40.994740 17347 catalog_manager.cc:5484] T a5e03cf44eab41cc842342a138c04067 P 29806d3d37354c169484f6db368f8798 reported cstate change: config changed from index 5 to 6, VOTER 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193) evicted. New cstate: current_term: 1 leader_uuid: "da3584f429104d23a1a8899b35cc2138" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } }
I20240502 01:54:41.013332 17478 tablet_service.cc:1514] Processing DeleteTablet for tablet 91e0e82526c0489a8908e250bf3c851f with delete_type TABLET_DATA_TOMBSTONED (TS 311f765f59754bdcb4eadc04e14bd4ea not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:48974
I20240502 01:54:41.020064 21766 tablet_replica.cc:330] T 91e0e82526c0489a8908e250bf3c851f P 311f765f59754bdcb4eadc04e14bd4ea: stopping tablet replica
I20240502 01:54:41.020325 21771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05b67d9912f047bfb6a2bed90b82ae15. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:41.022548 21766 raft_consensus.cc:2229] T 91e0e82526c0489a8908e250bf3c851f P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:41.022581 19869 raft_consensus.cc:2932] T 91e0e82526c0489a8908e250bf3c851f P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
I20240502 01:54:41.033469 21798 ts_tablet_manager.cc:902] T 3b3fa060e8d64911bec810ec07a358d2 P da3584f429104d23a1a8899b35cc2138: Initiating tablet copy from peer 1b50a031f2784ba498b3e52e16916205 (127.9.176.194:45617)
I20240502 01:54:41.042101 21766 raft_consensus.cc:2258] T 91e0e82526c0489a8908e250bf3c851f P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:41.042208 17478 tablet_service.cc:1514] Processing DeleteTablet for tablet a5e03cf44eab41cc842342a138c04067 with delete_type TABLET_DATA_TOMBSTONED (TS 311f765f59754bdcb4eadc04e14bd4ea not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:48974
I20240502 01:54:41.048311 17340 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a5e03cf44eab41cc842342a138c04067 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:41.061458 21797 ts_tablet_manager.cc:1888] T 5f3f3e5f772b4566b69236fd87d71a38 P 84d3a4b174c34e02aedab1e620731359: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:54:41.061762 21797 log.cc:1196] T 5f3f3e5f772b4566b69236fd87d71a38 P 84d3a4b174c34e02aedab1e620731359: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/wal/wals/5f3f3e5f772b4566b69236fd87d71a38
I20240502 01:54:41.063002 17343 catalog_manager.cc:4830] TS 84d3a4b174c34e02aedab1e620731359 (127.9.176.195:37357): tablet 5f3f3e5f772b4566b69236fd87d71a38 (table rebalance_test_table_7 [id=1bd69df0513c4c4db4dd6d6526efee4b]) successfully deleted
I20240502 01:54:41.063884 21787 ts_tablet_manager.cc:1888] T 075ded9523e44c3d92844805338643a3 P da3584f429104d23a1a8899b35cc2138: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:54:41.063797 21771 tablet_copy_client.cc:783] T 05b67d9912f047bfb6a2bed90b82ae15 P 94931f4b51c9493cb3d4264ce99f5e76: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:41.064352 21787 log.cc:1196] T 075ded9523e44c3d92844805338643a3 P da3584f429104d23a1a8899b35cc2138: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/wal/wals/075ded9523e44c3d92844805338643a3
I20240502 01:54:41.065007 21771 tablet_copy_client.cc:647] T 05b67d9912f047bfb6a2bed90b82ae15 P 94931f4b51c9493cb3d4264ce99f5e76: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:41.066030 17340 catalog_manager.cc:4830] TS da3584f429104d23a1a8899b35cc2138 (127.9.176.196:46381): tablet 075ded9523e44c3d92844805338643a3 (table rebalance_test_table_6 [id=d49bf19a92cf45ceadf02d01dbdae6ad]) successfully deleted
I20240502 01:54:41.071557 21771 tablet_copy_client.cc:523] T 05b67d9912f047bfb6a2bed90b82ae15 P 94931f4b51c9493cb3d4264ce99f5e76: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:41.075629 21771 tablet_bootstrap.cc:492] T 05b67d9912f047bfb6a2bed90b82ae15 P 94931f4b51c9493cb3d4264ce99f5e76: Bootstrap starting.
I20240502 01:54:41.082820 21766 ts_tablet_manager.cc:1875] T 91e0e82526c0489a8908e250bf3c851f P 311f765f59754bdcb4eadc04e14bd4ea: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:41.100663 21741 tablet_bootstrap.cc:492] T c91f74f46b04425aa5393515cb9f781c P 4b7c382c9edd41babdd1a1afc904a813: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:41.101186 21741 tablet_bootstrap.cc:492] T c91f74f46b04425aa5393515cb9f781c P 4b7c382c9edd41babdd1a1afc904a813: Bootstrap complete.
I20240502 01:54:41.101536 21741 ts_tablet_manager.cc:1367] T c91f74f46b04425aa5393515cb9f781c P 4b7c382c9edd41babdd1a1afc904a813: Time spent bootstrapping tablet: real 0.433s	user 0.044s	sys 0.014s
I20240502 01:54:41.103309 21741 raft_consensus.cc:348] T c91f74f46b04425aa5393515cb9f781c P 4b7c382c9edd41babdd1a1afc904a813 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } }
I20240502 01:54:41.103718 21741 raft_consensus.cc:729] T c91f74f46b04425aa5393515cb9f781c P 4b7c382c9edd41babdd1a1afc904a813 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4b7c382c9edd41babdd1a1afc904a813, State: Initialized, Role: LEARNER
I20240502 01:54:41.103966 21741 consensus_queue.cc:260] T c91f74f46b04425aa5393515cb9f781c P 4b7c382c9edd41babdd1a1afc904a813 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } }
I20240502 01:54:41.105433 21741 ts_tablet_manager.cc:1398] T c91f74f46b04425aa5393515cb9f781c P 4b7c382c9edd41babdd1a1afc904a813: Time spent starting tablet: real 0.004s	user 0.000s	sys 0.002s
I20240502 01:54:41.108531 17789 tablet_copy_service.cc:342] P 84d3a4b174c34e02aedab1e620731359: Request end of tablet copy session 4b7c382c9edd41babdd1a1afc904a813-c91f74f46b04425aa5393515cb9f781c received from {username='slave'} at 127.9.176.198:55177
I20240502 01:54:41.108776 17789 tablet_copy_service.cc:433] P 84d3a4b174c34e02aedab1e620731359: ending tablet copy session 4b7c382c9edd41babdd1a1afc904a813-c91f74f46b04425aa5393515cb9f781c on tablet c91f74f46b04425aa5393515cb9f781c with peer 4b7c382c9edd41babdd1a1afc904a813
I20240502 01:54:41.126114 17634 consensus_queue.cc:237] T 3b3fa060e8d64911bec810ec07a358d2 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { promote: true } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } }
I20240502 01:54:41.130021 17500 raft_consensus.cc:1261] T 3b3fa060e8d64911bec810ec07a358d2 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:41.130457 19736 raft_consensus.cc:1261] T 3b3fa060e8d64911bec810ec07a358d2 P 272b2179c1b4476090a89880df2a8baa [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:41.131286 21630 consensus_queue.cc:1035] T 3b3fa060e8d64911bec810ec07a358d2 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240502 01:54:41.131557 17565 consensus_peers.cc:485] T 3b3fa060e8d64911bec810ec07a358d2 P 1b50a031f2784ba498b3e52e16916205 -> Peer da3584f429104d23a1a8899b35cc2138 (127.9.176.196:46381): Couldn't send request to peer da3584f429104d23a1a8899b35cc2138. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: SHUTDOWN. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:41.131809 21708 consensus_queue.cc:1035] T 3b3fa060e8d64911bec810ec07a358d2 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240502 01:54:41.133270 17565 consensus_peers.cc:485] T 3b3fa060e8d64911bec810ec07a358d2 P 1b50a031f2784ba498b3e52e16916205 -> Peer 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198:36865): Couldn't send request to peer 4b7c382c9edd41babdd1a1afc904a813. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3b3fa060e8d64911bec810ec07a358d2. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:41.134851 21630 raft_consensus.cc:2932] T 3b3fa060e8d64911bec810ec07a358d2 P 1b50a031f2784ba498b3e52e16916205 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { promote: true } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } } }
I20240502 01:54:41.138072 19736 raft_consensus.cc:2932] T 3b3fa060e8d64911bec810ec07a358d2 P 272b2179c1b4476090a89880df2a8baa [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { promote: true } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } } }
I20240502 01:54:41.141383 17500 raft_consensus.cc:2932] T 3b3fa060e8d64911bec810ec07a358d2 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { promote: true } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } } }
I20240502 01:54:41.148407 17348 catalog_manager.cc:5484] T 3b3fa060e8d64911bec810ec07a358d2 P 272b2179c1b4476090a89880df2a8baa reported cstate change: config changed from index 7 to 8, NON_VOTER 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) added. New cstate: current_term: 1 leader_uuid: "1b50a031f2784ba498b3e52e16916205" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { promote: true } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } } }
I20240502 01:54:41.209854 17500 consensus_queue.cc:237] T 37eda67598fe491bbb29aa62b2c4bec7 P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { promote: true } }
I20240502 01:54:41.216461 21766 ts_tablet_manager.cc:1888] T 91e0e82526c0489a8908e250bf3c851f P 311f765f59754bdcb4eadc04e14bd4ea: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 01:54:41.216655 21766 log.cc:1196] T 91e0e82526c0489a8908e250bf3c851f P 311f765f59754bdcb4eadc04e14bd4ea: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/wal/wals/91e0e82526c0489a8908e250bf3c851f
I20240502 01:54:41.217797 17343 catalog_manager.cc:4830] TS 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193:35397): tablet 91e0e82526c0489a8908e250bf3c851f (table rebalance_test_table_2 [id=bf67e64a43cd4367a7420c6bf1793faa]) successfully deleted
I20240502 01:54:41.218837 21766 tablet_replica.cc:330] T a5e03cf44eab41cc842342a138c04067 P 311f765f59754bdcb4eadc04e14bd4ea: stopping tablet replica
I20240502 01:54:41.219136 21766 raft_consensus.cc:2229] T a5e03cf44eab41cc842342a138c04067 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:41.219494 21766 raft_consensus.cc:2258] T a5e03cf44eab41cc842342a138c04067 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:41.221503 21766 ts_tablet_manager.cc:1875] T a5e03cf44eab41cc842342a138c04067 P 311f765f59754bdcb4eadc04e14bd4ea: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240502 01:54:41.229663 17432 consensus_peers.cc:485] T 37eda67598fe491bbb29aa62b2c4bec7 P 311f765f59754bdcb4eadc04e14bd4ea -> Peer da3584f429104d23a1a8899b35cc2138 (127.9.176.196:46381): Couldn't send request to peer da3584f429104d23a1a8899b35cc2138. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 37eda67598fe491bbb29aa62b2c4bec7. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:41.230453 19467 raft_consensus.cc:1261] T 37eda67598fe491bbb29aa62b2c4bec7 P 29806d3d37354c169484f6db368f8798 [term 1 FOLLOWER]: Refusing update from remote peer 311f765f59754bdcb4eadc04e14bd4ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 01:54:41.230509 19869 raft_consensus.cc:1261] T 37eda67598fe491bbb29aa62b2c4bec7 P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Refusing update from remote peer 311f765f59754bdcb4eadc04e14bd4ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 01:54:41.233017 21467 consensus_queue.cc:1035] T 37eda67598fe491bbb29aa62b2c4bec7 P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 01:54:41.233793 21467 consensus_queue.cc:1035] T 37eda67598fe491bbb29aa62b2c4bec7 P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 01:54:41.247624 21766 ts_tablet_manager.cc:1888] T a5e03cf44eab41cc842342a138c04067 P 311f765f59754bdcb4eadc04e14bd4ea: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:54:41.247808 21766 log.cc:1196] T a5e03cf44eab41cc842342a138c04067 P 311f765f59754bdcb4eadc04e14bd4ea: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/wal/wals/a5e03cf44eab41cc842342a138c04067
I20240502 01:54:41.250602 17343 catalog_manager.cc:4830] TS 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193:35397): tablet a5e03cf44eab41cc842342a138c04067 (table rebalance_test_table_4 [id=14b25b3f4ede4b65870815b014d90449]) successfully deleted
I20240502 01:54:41.254441 21734 raft_consensus.cc:2932] T 37eda67598fe491bbb29aa62b2c4bec7 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER da3584f429104d23a1a8899b35cc2138 (127.9.176.196) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { promote: true } } }
I20240502 01:54:41.265183 21798 tablet_copy_client.cc:308] T 3b3fa060e8d64911bec810ec07a358d2 P da3584f429104d23a1a8899b35cc2138: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.194:45617
I20240502 01:54:41.265750 19869 raft_consensus.cc:2932] T 37eda67598fe491bbb29aa62b2c4bec7 P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER da3584f429104d23a1a8899b35cc2138 (127.9.176.196) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { promote: true } } }
I20240502 01:54:41.266307 17654 tablet_copy_service.cc:140] P 1b50a031f2784ba498b3e52e16916205: Received BeginTabletCopySession request for tablet 3b3fa060e8d64911bec810ec07a358d2 from peer da3584f429104d23a1a8899b35cc2138 ({username='slave'} at 127.9.176.196:38229)
I20240502 01:54:41.266510 17654 tablet_copy_service.cc:161] P 1b50a031f2784ba498b3e52e16916205: Beginning new tablet copy session on tablet 3b3fa060e8d64911bec810ec07a358d2 from peer da3584f429104d23a1a8899b35cc2138 at {username='slave'} at 127.9.176.196:38229: session id = da3584f429104d23a1a8899b35cc2138-3b3fa060e8d64911bec810ec07a358d2
I20240502 01:54:41.269476 17654 tablet_copy_source_session.cc:205] T 3b3fa060e8d64911bec810ec07a358d2 P 1b50a031f2784ba498b3e52e16916205: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:41.282754 21798 ts_tablet_manager.cc:1875] T 3b3fa060e8d64911bec810ec07a358d2 P da3584f429104d23a1a8899b35cc2138: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 01:54:41.294392 17351 catalog_manager.cc:5484] T 37eda67598fe491bbb29aa62b2c4bec7 P 94931f4b51c9493cb3d4264ce99f5e76 reported cstate change: config changed from index 11 to 12, NON_VOTER da3584f429104d23a1a8899b35cc2138 (127.9.176.196) added. New cstate: current_term: 1 leader_uuid: "311f765f59754bdcb4eadc04e14bd4ea" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { promote: true } } }
I20240502 01:54:41.305773 17613 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:54:41.329813 21842 ts_tablet_manager.cc:902] T 6da59c0626c64a5988e39a831c42c268 P 272b2179c1b4476090a89880df2a8baa: Initiating tablet copy from peer 84d3a4b174c34e02aedab1e620731359 (127.9.176.195:37357)
I20240502 01:54:41.334295 21844 ts_tablet_manager.cc:902] T 2c78951695284b1aaab411c0a041c28a P 29806d3d37354c169484f6db368f8798: Initiating tablet copy from peer 1b50a031f2784ba498b3e52e16916205 (127.9.176.194:45617)
I20240502 01:54:41.357163 19467 raft_consensus.cc:2932] T 37eda67598fe491bbb29aa62b2c4bec7 P 29806d3d37354c169484f6db368f8798 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER da3584f429104d23a1a8899b35cc2138 (127.9.176.196) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { promote: true } } }
I20240502 01:54:41.377286 17764 consensus_queue.cc:237] T 2045e08a200148c69326351f38fd484d P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { promote: true } }
I20240502 01:54:41.389256 21741 ts_tablet_manager.cc:902] T f07e9b433e554c85b3c25b7e708f8b2f P 4b7c382c9edd41babdd1a1afc904a813: Initiating tablet copy from peer 84d3a4b174c34e02aedab1e620731359 (127.9.176.195:37357)
I20240502 01:54:41.391644 21844 tablet_copy_client.cc:308] T 2c78951695284b1aaab411c0a041c28a P 29806d3d37354c169484f6db368f8798: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.194:45617
I20240502 01:54:41.392575 17654 tablet_copy_service.cc:140] P 1b50a031f2784ba498b3e52e16916205: Received BeginTabletCopySession request for tablet 2c78951695284b1aaab411c0a041c28a from peer 29806d3d37354c169484f6db368f8798 ({username='slave'} at 127.9.176.197:33725)
I20240502 01:54:41.392822 17654 tablet_copy_service.cc:161] P 1b50a031f2784ba498b3e52e16916205: Beginning new tablet copy session on tablet 2c78951695284b1aaab411c0a041c28a from peer 29806d3d37354c169484f6db368f8798 at {username='slave'} at 127.9.176.197:33725: session id = 29806d3d37354c169484f6db368f8798-2c78951695284b1aaab411c0a041c28a
I20240502 01:54:41.396097 17654 tablet_copy_source_session.cc:205] T 2c78951695284b1aaab411c0a041c28a P 1b50a031f2784ba498b3e52e16916205: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:41.398844 21798 ts_tablet_manager.cc:1888] T 3b3fa060e8d64911bec810ec07a358d2 P da3584f429104d23a1a8899b35cc2138: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240502 01:54:41.399329 21798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b3fa060e8d64911bec810ec07a358d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:41.411841 19603 raft_consensus.cc:1203] T c91f74f46b04425aa5393515cb9f781c P 4b7c382c9edd41babdd1a1afc904a813 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240502 01:54:41.411991 17478 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:54:41.412622 21842 tablet_copy_client.cc:308] T 6da59c0626c64a5988e39a831c42c268 P 272b2179c1b4476090a89880df2a8baa: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.195:37357
I20240502 01:54:41.413403 17789 tablet_copy_service.cc:140] P 84d3a4b174c34e02aedab1e620731359: Received BeginTabletCopySession request for tablet 6da59c0626c64a5988e39a831c42c268 from peer 272b2179c1b4476090a89880df2a8baa ({username='slave'} at 127.9.176.199:46159)
I20240502 01:54:41.413589 17789 tablet_copy_service.cc:161] P 84d3a4b174c34e02aedab1e620731359: Beginning new tablet copy session on tablet 6da59c0626c64a5988e39a831c42c268 from peer 272b2179c1b4476090a89880df2a8baa at {username='slave'} at 127.9.176.199:46159: session id = 272b2179c1b4476090a89880df2a8baa-6da59c0626c64a5988e39a831c42c268
I20240502 01:54:41.419250 17789 tablet_copy_source_session.cc:205] T 6da59c0626c64a5988e39a831c42c268 P 84d3a4b174c34e02aedab1e620731359: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:41.423645 21844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c78951695284b1aaab411c0a041c28a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:41.425115 21842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6da59c0626c64a5988e39a831c42c268. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:41.434453 19718 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:41.445135 19467 raft_consensus.cc:1261] T 2045e08a200148c69326351f38fd484d P 29806d3d37354c169484f6db368f8798 [term 1 FOLLOWER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 01:54:41.445761 17700 consensus_peers.cc:485] T 2045e08a200148c69326351f38fd484d P 84d3a4b174c34e02aedab1e620731359 -> Peer da3584f429104d23a1a8899b35cc2138 (127.9.176.196:46381): Couldn't send request to peer da3584f429104d23a1a8899b35cc2138. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:41.447279 17634 raft_consensus.cc:1261] T 2045e08a200148c69326351f38fd484d P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:41.448997 21776 consensus_queue.cc:1035] T 2045e08a200148c69326351f38fd484d P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:41.462623 19446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:41.463059 21362 consensus_queue.cc:1035] T 2045e08a200148c69326351f38fd484d P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:41.471017 21362 raft_consensus.cc:2932] T 2045e08a200148c69326351f38fd484d P 84d3a4b174c34e02aedab1e620731359 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER da3584f429104d23a1a8899b35cc2138 (127.9.176.196) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { promote: true } } }
I20240502 01:54:41.477102 17631 raft_consensus.cc:2932] T 2045e08a200148c69326351f38fd484d P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER da3584f429104d23a1a8899b35cc2138 (127.9.176.196) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { promote: true } } }
I20240502 01:54:41.477700 17746 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:54:41.478037 21798 tablet_copy_client.cc:783] T 3b3fa060e8d64911bec810ec07a358d2 P da3584f429104d23a1a8899b35cc2138: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:41.478602 21798 tablet_copy_client.cc:647] T 3b3fa060e8d64911bec810ec07a358d2 P da3584f429104d23a1a8899b35cc2138: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:41.481266 21798 tablet_copy_client.cc:523] T 3b3fa060e8d64911bec810ec07a358d2 P da3584f429104d23a1a8899b35cc2138: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:41.486282 17348 catalog_manager.cc:5484] T 2045e08a200148c69326351f38fd484d P 1b50a031f2784ba498b3e52e16916205 reported cstate change: config changed from index 6 to 7, NON_VOTER da3584f429104d23a1a8899b35cc2138 (127.9.176.196) added. New cstate: current_term: 1 leader_uuid: "84d3a4b174c34e02aedab1e620731359" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { promote: true } } }
I20240502 01:54:41.489753 19585 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:41.516333 19466 raft_consensus.cc:2932] T 2045e08a200148c69326351f38fd484d P 29806d3d37354c169484f6db368f8798 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER da3584f429104d23a1a8899b35cc2138 (127.9.176.196) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { promote: true } } }
I20240502 01:54:41.534761 19852 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:41.547737 21882 ts_tablet_manager.cc:902] T 3b3fa060e8d64911bec810ec07a358d2 P 4b7c382c9edd41babdd1a1afc904a813: Initiating tablet copy from peer 1b50a031f2784ba498b3e52e16916205 (127.9.176.194:45617)
I20240502 01:54:41.568672 21844 tablet_copy_client.cc:783] T 2c78951695284b1aaab411c0a041c28a P 29806d3d37354c169484f6db368f8798: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:41.569103 21844 tablet_copy_client.cc:647] T 2c78951695284b1aaab411c0a041c28a P 29806d3d37354c169484f6db368f8798: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:41.571875 21844 tablet_copy_client.cc:523] T 2c78951695284b1aaab411c0a041c28a P 29806d3d37354c169484f6db368f8798: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:41.576743 21798 tablet_bootstrap.cc:492] T 3b3fa060e8d64911bec810ec07a358d2 P da3584f429104d23a1a8899b35cc2138: Bootstrap starting.
I20240502 01:54:41.579712 21842 tablet_copy_client.cc:783] T 6da59c0626c64a5988e39a831c42c268 P 272b2179c1b4476090a89880df2a8baa: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:41.580116 21842 tablet_copy_client.cc:647] T 6da59c0626c64a5988e39a831c42c268 P 272b2179c1b4476090a89880df2a8baa: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:41.582756 21842 tablet_copy_client.cc:523] T 6da59c0626c64a5988e39a831c42c268 P 272b2179c1b4476090a89880df2a8baa: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:41.585548 21741 tablet_copy_client.cc:308] T f07e9b433e554c85b3c25b7e708f8b2f P 4b7c382c9edd41babdd1a1afc904a813: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.195:37357
I20240502 01:54:41.586753 17789 tablet_copy_service.cc:140] P 84d3a4b174c34e02aedab1e620731359: Received BeginTabletCopySession request for tablet f07e9b433e554c85b3c25b7e708f8b2f from peer 4b7c382c9edd41babdd1a1afc904a813 ({username='slave'} at 127.9.176.198:55177)
I20240502 01:54:41.587008 17789 tablet_copy_service.cc:161] P 84d3a4b174c34e02aedab1e620731359: Beginning new tablet copy session on tablet f07e9b433e554c85b3c25b7e708f8b2f from peer 4b7c382c9edd41babdd1a1afc904a813 at {username='slave'} at 127.9.176.198:55177: session id = 4b7c382c9edd41babdd1a1afc904a813-f07e9b433e554c85b3c25b7e708f8b2f
I20240502 01:54:41.590932 17789 tablet_copy_source_session.cc:205] T f07e9b433e554c85b3c25b7e708f8b2f P 84d3a4b174c34e02aedab1e620731359: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:41.592972 21741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f07e9b433e554c85b3c25b7e708f8b2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:41.594033 17880 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:54:41.615693 21741 tablet_copy_client.cc:783] T f07e9b433e554c85b3c25b7e708f8b2f P 4b7c382c9edd41babdd1a1afc904a813: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:41.616130 21741 tablet_copy_client.cc:647] T f07e9b433e554c85b3c25b7e708f8b2f P 4b7c382c9edd41babdd1a1afc904a813: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:41.620739 21741 tablet_copy_client.cc:523] T f07e9b433e554c85b3c25b7e708f8b2f P 4b7c382c9edd41babdd1a1afc904a813: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:41.629568 21844 tablet_bootstrap.cc:492] T 2c78951695284b1aaab411c0a041c28a P 29806d3d37354c169484f6db368f8798: Bootstrap starting.
I20240502 01:54:41.630458 21842 tablet_bootstrap.cc:492] T 6da59c0626c64a5988e39a831c42c268 P 272b2179c1b4476090a89880df2a8baa: Bootstrap starting.
I20240502 01:54:41.632211 21771 tablet_bootstrap.cc:492] T 05b67d9912f047bfb6a2bed90b82ae15 P 94931f4b51c9493cb3d4264ce99f5e76: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:41.635479 21771 tablet_bootstrap.cc:492] T 05b67d9912f047bfb6a2bed90b82ae15 P 94931f4b51c9493cb3d4264ce99f5e76: Bootstrap complete.
I20240502 01:54:41.638736 21771 ts_tablet_manager.cc:1367] T 05b67d9912f047bfb6a2bed90b82ae15 P 94931f4b51c9493cb3d4264ce99f5e76: Time spent bootstrapping tablet: real 0.563s	user 0.056s	sys 0.016s
I20240502 01:54:41.643451 21741 tablet_bootstrap.cc:492] T f07e9b433e554c85b3c25b7e708f8b2f P 4b7c382c9edd41babdd1a1afc904a813: Bootstrap starting.
I20240502 01:54:41.646636 21771 raft_consensus.cc:348] T 05b67d9912f047bfb6a2bed90b82ae15 P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } }
I20240502 01:54:41.646981 21771 raft_consensus.cc:729] T 05b67d9912f047bfb6a2bed90b82ae15 P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 94931f4b51c9493cb3d4264ce99f5e76, State: Initialized, Role: LEARNER
I20240502 01:54:41.647224 21771 consensus_queue.cc:260] T 05b67d9912f047bfb6a2bed90b82ae15 P 94931f4b51c9493cb3d4264ce99f5e76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } }
I20240502 01:54:41.653050 21771 ts_tablet_manager.cc:1398] T 05b67d9912f047bfb6a2bed90b82ae15 P 94931f4b51c9493cb3d4264ce99f5e76: Time spent starting tablet: real 0.014s	user 0.003s	sys 0.000s
I20240502 01:54:41.655615 17922 tablet_copy_service.cc:342] P da3584f429104d23a1a8899b35cc2138: Request end of tablet copy session 94931f4b51c9493cb3d4264ce99f5e76-05b67d9912f047bfb6a2bed90b82ae15 received from {username='slave'} at 127.9.176.200:36935
I20240502 01:54:41.655810 17922 tablet_copy_service.cc:433] P da3584f429104d23a1a8899b35cc2138: ending tablet copy session 94931f4b51c9493cb3d4264ce99f5e76-05b67d9912f047bfb6a2bed90b82ae15 on tablet 05b67d9912f047bfb6a2bed90b82ae15 with peer 94931f4b51c9493cb3d4264ce99f5e76
I20240502 01:54:41.669533 17500 consensus_queue.cc:237] T 49ef898a43704b87ba077cd7b0485e9a P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } }
I20240502 01:54:41.670485 17901 consensus_queue.cc:237] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P da3584f429104d23a1a8899b35cc2138 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: NON_VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { promote: true } }
I20240502 01:54:41.690181 19869 raft_consensus.cc:1261] T 49ef898a43704b87ba077cd7b0485e9a P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Refusing update from remote peer 311f765f59754bdcb4eadc04e14bd4ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:41.691181 21734 consensus_queue.cc:1035] T 49ef898a43704b87ba077cd7b0485e9a P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240502 01:54:41.693418 17835 consensus_peers.cc:485] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P da3584f429104d23a1a8899b35cc2138 -> Peer 84d3a4b174c34e02aedab1e620731359 (127.9.176.195:37357): Couldn't send request to peer 84d3a4b174c34e02aedab1e620731359. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:41.693936 19736 raft_consensus.cc:1261] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 272b2179c1b4476090a89880df2a8baa [term 1 FOLLOWER]: Refusing update from remote peer da3584f429104d23a1a8899b35cc2138: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:54:41.694624 21778 consensus_queue.cc:1035] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P da3584f429104d23a1a8899b35cc2138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:54:41.699759 17634 raft_consensus.cc:1261] T 49ef898a43704b87ba077cd7b0485e9a P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Refusing update from remote peer 311f765f59754bdcb4eadc04e14bd4ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:41.704154 21734 consensus_queue.cc:1035] T 49ef898a43704b87ba077cd7b0485e9a P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:41.711551 19870 raft_consensus.cc:1203] T 05b67d9912f047bfb6a2bed90b82ae15 P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240502 01:54:41.742082 17898 raft_consensus.cc:1261] T 49ef898a43704b87ba077cd7b0485e9a P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Refusing update from remote peer 311f765f59754bdcb4eadc04e14bd4ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:41.742971 21734 consensus_queue.cc:1035] T 49ef898a43704b87ba077cd7b0485e9a P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:41.754552 19869 raft_consensus.cc:1261] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Refusing update from remote peer da3584f429104d23a1a8899b35cc2138: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:54:41.755383 20915 consensus_queue.cc:1035] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P da3584f429104d23a1a8899b35cc2138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:54:41.764731 20915 raft_consensus.cc:2932] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P da3584f429104d23a1a8899b35cc2138 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 84d3a4b174c34e02aedab1e620731359 (127.9.176.195) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: NON_VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { promote: true } } }
I20240502 01:54:41.771039 19870 raft_consensus.cc:2932] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 84d3a4b174c34e02aedab1e620731359 (127.9.176.195) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: NON_VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { promote: true } } }
I20240502 01:54:41.773104 21741 tablet_bootstrap.cc:492] T f07e9b433e554c85b3c25b7e708f8b2f P 4b7c382c9edd41babdd1a1afc904a813: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:41.773662 21741 tablet_bootstrap.cc:492] T f07e9b433e554c85b3c25b7e708f8b2f P 4b7c382c9edd41babdd1a1afc904a813: Bootstrap complete.
I20240502 01:54:41.774034 21741 ts_tablet_manager.cc:1367] T f07e9b433e554c85b3c25b7e708f8b2f P 4b7c382c9edd41babdd1a1afc904a813: Time spent bootstrapping tablet: real 0.131s	user 0.070s	sys 0.001s
I20240502 01:54:41.775132 21741 raft_consensus.cc:348] T f07e9b433e554c85b3c25b7e708f8b2f P 4b7c382c9edd41babdd1a1afc904a813 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } }
I20240502 01:54:41.775491 21741 raft_consensus.cc:729] T f07e9b433e554c85b3c25b7e708f8b2f P 4b7c382c9edd41babdd1a1afc904a813 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4b7c382c9edd41babdd1a1afc904a813, State: Initialized, Role: LEARNER
I20240502 01:54:41.775805 21741 consensus_queue.cc:260] T f07e9b433e554c85b3c25b7e708f8b2f P 4b7c382c9edd41babdd1a1afc904a813 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } }
I20240502 01:54:41.779696 21741 ts_tablet_manager.cc:1398] T f07e9b433e554c85b3c25b7e708f8b2f P 4b7c382c9edd41babdd1a1afc904a813: Time spent starting tablet: real 0.006s	user 0.000s	sys 0.002s
I20240502 01:54:41.780545 17789 tablet_copy_service.cc:342] P 84d3a4b174c34e02aedab1e620731359: Request end of tablet copy session 4b7c382c9edd41babdd1a1afc904a813-f07e9b433e554c85b3c25b7e708f8b2f received from {username='slave'} at 127.9.176.198:55177
I20240502 01:54:41.780699 17789 tablet_copy_service.cc:433] P 84d3a4b174c34e02aedab1e620731359: ending tablet copy session 4b7c382c9edd41babdd1a1afc904a813-f07e9b433e554c85b3c25b7e708f8b2f on tablet f07e9b433e554c85b3c25b7e708f8b2f with peer 4b7c382c9edd41babdd1a1afc904a813
I20240502 01:54:41.781406 17348 catalog_manager.cc:5484] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 94931f4b51c9493cb3d4264ce99f5e76 reported cstate change: config changed from index 9 to 10, NON_VOTER 84d3a4b174c34e02aedab1e620731359 (127.9.176.195) added. New cstate: current_term: 1 leader_uuid: "da3584f429104d23a1a8899b35cc2138" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: NON_VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { promote: true } } }
I20240502 01:54:41.804725 21734 raft_consensus.cc:2932] T 49ef898a43704b87ba077cd7b0485e9a P 311f765f59754bdcb4eadc04e14bd4ea [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
I20240502 01:54:41.808691 17631 raft_consensus.cc:2932] T 49ef898a43704b87ba077cd7b0485e9a P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
I20240502 01:54:41.821702 19869 raft_consensus.cc:2932] T 49ef898a43704b87ba077cd7b0485e9a P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
I20240502 01:54:41.838362 21882 tablet_copy_client.cc:308] T 3b3fa060e8d64911bec810ec07a358d2 P 4b7c382c9edd41babdd1a1afc904a813: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.194:45617
I20240502 01:54:41.840533 17347 catalog_manager.cc:5484] T 49ef898a43704b87ba077cd7b0485e9a P 1b50a031f2784ba498b3e52e16916205 reported cstate change: config changed from index 5 to 6. New cstate: current_term: 1 leader_uuid: "311f765f59754bdcb4eadc04e14bd4ea" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
I20240502 01:54:41.840677 19736 raft_consensus.cc:2932] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 272b2179c1b4476090a89880df2a8baa [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 84d3a4b174c34e02aedab1e620731359 (127.9.176.195) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: NON_VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { promote: true } } }
I20240502 01:54:41.847985 21907 ts_tablet_manager.cc:902] T 2045e08a200148c69326351f38fd484d P da3584f429104d23a1a8899b35cc2138: Initiating tablet copy from peer 84d3a4b174c34e02aedab1e620731359 (127.9.176.195:37357)
I20240502 01:54:41.856072 17654 tablet_copy_service.cc:140] P 1b50a031f2784ba498b3e52e16916205: Received BeginTabletCopySession request for tablet 3b3fa060e8d64911bec810ec07a358d2 from peer 4b7c382c9edd41babdd1a1afc904a813 ({username='slave'} at 127.9.176.198:49387)
I20240502 01:54:41.856313 17654 tablet_copy_service.cc:161] P 1b50a031f2784ba498b3e52e16916205: Beginning new tablet copy session on tablet 3b3fa060e8d64911bec810ec07a358d2 from peer 4b7c382c9edd41babdd1a1afc904a813 at {username='slave'} at 127.9.176.198:49387: session id = 4b7c382c9edd41babdd1a1afc904a813-3b3fa060e8d64911bec810ec07a358d2
I20240502 01:54:41.859758 17654 tablet_copy_source_session.cc:205] T 3b3fa060e8d64911bec810ec07a358d2 P 1b50a031f2784ba498b3e52e16916205: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:41.863873 21882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b3fa060e8d64911bec810ec07a358d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:41.860783 17897 raft_consensus.cc:2932] T 49ef898a43704b87ba077cd7b0485e9a P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
I20240502 01:54:41.899533 21882 tablet_copy_client.cc:783] T 3b3fa060e8d64911bec810ec07a358d2 P 4b7c382c9edd41babdd1a1afc904a813: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:41.900213 21882 tablet_copy_client.cc:647] T 3b3fa060e8d64911bec810ec07a358d2 P 4b7c382c9edd41babdd1a1afc904a813: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:41.903077 21882 tablet_copy_client.cc:523] T 3b3fa060e8d64911bec810ec07a358d2 P 4b7c382c9edd41babdd1a1afc904a813: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:41.910771 21930 ts_tablet_manager.cc:902] T 37eda67598fe491bbb29aa62b2c4bec7 P da3584f429104d23a1a8899b35cc2138: Initiating tablet copy from peer 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193:35397)
I20240502 01:54:41.912202 17500 consensus_queue.cc:237] T 49ef898a43704b87ba077cd7b0485e9a P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } }
W20240502 01:54:41.915078 17432 consensus_peers.cc:485] T 49ef898a43704b87ba077cd7b0485e9a P 311f765f59754bdcb4eadc04e14bd4ea -> Peer 29806d3d37354c169484f6db368f8798 (127.9.176.197:35011): Couldn't send request to peer 29806d3d37354c169484f6db368f8798. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 49ef898a43704b87ba077cd7b0485e9a. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:41.915823 21362 raft_consensus.cc:1050] T c91f74f46b04425aa5393515cb9f781c P 84d3a4b174c34e02aedab1e620731359: attempting to promote NON_VOTER 4b7c382c9edd41babdd1a1afc904a813 to VOTER
I20240502 01:54:41.916713 21362 consensus_queue.cc:237] T c91f74f46b04425aa5393515cb9f781c P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } }
I20240502 01:54:41.917234 17634 raft_consensus.cc:1261] T 49ef898a43704b87ba077cd7b0485e9a P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Refusing update from remote peer 311f765f59754bdcb4eadc04e14bd4ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:41.918054 21467 consensus_queue.cc:1035] T 49ef898a43704b87ba077cd7b0485e9a P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:41.921173 19870 raft_consensus.cc:1261] T 49ef898a43704b87ba077cd7b0485e9a P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Refusing update from remote peer 311f765f59754bdcb4eadc04e14bd4ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:41.922425 21467 consensus_queue.cc:1035] T 49ef898a43704b87ba077cd7b0485e9a P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:41.925995 21467 raft_consensus.cc:2932] T 49ef898a43704b87ba077cd7b0485e9a P 311f765f59754bdcb4eadc04e14bd4ea [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 29806d3d37354c169484f6db368f8798 (127.9.176.197) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } } }
I20240502 01:54:41.928185 19872 raft_consensus.cc:2932] T 49ef898a43704b87ba077cd7b0485e9a P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 29806d3d37354c169484f6db368f8798 (127.9.176.197) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } } }
I20240502 01:54:41.935988 17901 raft_consensus.cc:1261] T 49ef898a43704b87ba077cd7b0485e9a P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Refusing update from remote peer 311f765f59754bdcb4eadc04e14bd4ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:41.936966 19603 raft_consensus.cc:1261] T c91f74f46b04425aa5393515cb9f781c P 4b7c382c9edd41babdd1a1afc904a813 [term 1 LEARNER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:41.938323 21734 consensus_queue.cc:1035] T 49ef898a43704b87ba077cd7b0485e9a P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:41.938288 21776 consensus_queue.cc:1035] T c91f74f46b04425aa5393515cb9f781c P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:41.943128 17343 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 49ef898a43704b87ba077cd7b0485e9a with cas_config_opid_index 6: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 01:54:41.946882 19466 raft_consensus.cc:1261] T c91f74f46b04425aa5393515cb9f781c P 29806d3d37354c169484f6db368f8798 [term 1 FOLLOWER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:41.945361 17901 raft_consensus.cc:2932] T 49ef898a43704b87ba077cd7b0485e9a P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 29806d3d37354c169484f6db368f8798 (127.9.176.197) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } } }
I20240502 01:54:41.947878 21776 consensus_queue.cc:1035] T c91f74f46b04425aa5393515cb9f781c P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:41.959911 17500 raft_consensus.cc:1261] T c91f74f46b04425aa5393515cb9f781c P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:41.960810 21776 consensus_queue.cc:1035] T c91f74f46b04425aa5393515cb9f781c P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:41.961697 21882 tablet_bootstrap.cc:492] T 3b3fa060e8d64911bec810ec07a358d2 P 4b7c382c9edd41babdd1a1afc904a813: Bootstrap starting.
I20240502 01:54:41.991818 21930 tablet_copy_client.cc:308] T 37eda67598fe491bbb29aa62b2c4bec7 P da3584f429104d23a1a8899b35cc2138: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.193:35397
I20240502 01:54:41.992786 17520 tablet_copy_service.cc:140] P 311f765f59754bdcb4eadc04e14bd4ea: Received BeginTabletCopySession request for tablet 37eda67598fe491bbb29aa62b2c4bec7 from peer da3584f429104d23a1a8899b35cc2138 ({username='slave'} at 127.9.176.196:38671)
I20240502 01:54:41.992972 17520 tablet_copy_service.cc:161] P 311f765f59754bdcb4eadc04e14bd4ea: Beginning new tablet copy session on tablet 37eda67598fe491bbb29aa62b2c4bec7 from peer da3584f429104d23a1a8899b35cc2138 at {username='slave'} at 127.9.176.196:38671: session id = da3584f429104d23a1a8899b35cc2138-37eda67598fe491bbb29aa62b2c4bec7
I20240502 01:54:41.994419 19604 raft_consensus.cc:1203] T f07e9b433e554c85b3c25b7e708f8b2f P 4b7c382c9edd41babdd1a1afc904a813 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240502 01:54:41.996927 17520 tablet_copy_source_session.cc:205] T 37eda67598fe491bbb29aa62b2c4bec7 P 311f765f59754bdcb4eadc04e14bd4ea: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:41.998759 17631 raft_consensus.cc:2932] T 49ef898a43704b87ba077cd7b0485e9a P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 29806d3d37354c169484f6db368f8798 (127.9.176.197) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } } }
I20240502 01:54:41.999517 21930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37eda67598fe491bbb29aa62b2c4bec7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:42.008888 21776 raft_consensus.cc:2932] T c91f74f46b04425aa5393515cb9f781c P 84d3a4b174c34e02aedab1e620731359 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
W20240502 01:54:42.013717 21934 log.cc:924] Time spent T c91f74f46b04425aa5393515cb9f781c P 84d3a4b174c34e02aedab1e620731359: Append to log took a long time: real 0.078s	user 0.000s	sys 0.000s
I20240502 01:54:42.016088 21798 tablet_bootstrap.cc:492] T 3b3fa060e8d64911bec810ec07a358d2 P da3584f429104d23a1a8899b35cc2138: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:42.016628 21798 tablet_bootstrap.cc:492] T 3b3fa060e8d64911bec810ec07a358d2 P da3584f429104d23a1a8899b35cc2138: Bootstrap complete.
I20240502 01:54:42.016378 19603 raft_consensus.cc:2932] T c91f74f46b04425aa5393515cb9f781c P 4b7c382c9edd41babdd1a1afc904a813 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
I20240502 01:54:42.016966 21798 ts_tablet_manager.cc:1367] T 3b3fa060e8d64911bec810ec07a358d2 P da3584f429104d23a1a8899b35cc2138: Time spent bootstrapping tablet: real 0.440s	user 0.065s	sys 0.000s
I20240502 01:54:42.017063 17352 catalog_manager.cc:5484] T c91f74f46b04425aa5393515cb9f781c P 84d3a4b174c34e02aedab1e620731359 reported cstate change: config changed from index 7 to 8, 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "84d3a4b174c34e02aedab1e620731359" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:42.018124 21798 raft_consensus.cc:348] T 3b3fa060e8d64911bec810ec07a358d2 P da3584f429104d23a1a8899b35cc2138 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { promote: true } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } }
I20240502 01:54:42.018461 21798 raft_consensus.cc:729] T 3b3fa060e8d64911bec810ec07a358d2 P da3584f429104d23a1a8899b35cc2138 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: da3584f429104d23a1a8899b35cc2138, State: Initialized, Role: LEARNER
I20240502 01:54:42.018692 21798 consensus_queue.cc:260] T 3b3fa060e8d64911bec810ec07a358d2 P da3584f429104d23a1a8899b35cc2138 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { promote: true } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } }
I20240502 01:54:42.019233 17500 raft_consensus.cc:2932] T c91f74f46b04425aa5393515cb9f781c P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
I20240502 01:54:42.023000 17764 consensus_queue.cc:237] T c91f74f46b04425aa5393515cb9f781c P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } }
I20240502 01:54:42.024682 17348 catalog_manager.cc:5484] T 49ef898a43704b87ba077cd7b0485e9a P 94931f4b51c9493cb3d4264ce99f5e76 reported cstate change: config changed from index 6 to 7, NON_VOTER 29806d3d37354c169484f6db368f8798 (127.9.176.197) added. New cstate: current_term: 1 leader_uuid: "311f765f59754bdcb4eadc04e14bd4ea" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } } }
I20240502 01:54:42.011096 21842 tablet_bootstrap.cc:492] T 6da59c0626c64a5988e39a831c42c268 P 272b2179c1b4476090a89880df2a8baa: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:42.028393 21842 tablet_bootstrap.cc:492] T 6da59c0626c64a5988e39a831c42c268 P 272b2179c1b4476090a89880df2a8baa: Bootstrap complete.
I20240502 01:54:42.028955 21930 tablet_copy_client.cc:783] T 37eda67598fe491bbb29aa62b2c4bec7 P da3584f429104d23a1a8899b35cc2138: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:42.029311 21930 tablet_copy_client.cc:647] T 37eda67598fe491bbb29aa62b2c4bec7 P da3584f429104d23a1a8899b35cc2138: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:42.028764 21842 ts_tablet_manager.cc:1367] T 6da59c0626c64a5988e39a831c42c268 P 272b2179c1b4476090a89880df2a8baa: Time spent bootstrapping tablet: real 0.398s	user 0.060s	sys 0.003s
I20240502 01:54:42.031026 19603 raft_consensus.cc:1261] T c91f74f46b04425aa5393515cb9f781c P 4b7c382c9edd41babdd1a1afc904a813 [term 1 FOLLOWER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:54:42.031689 21362 consensus_queue.cc:1035] T c91f74f46b04425aa5393515cb9f781c P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:42.036085 21930 tablet_copy_client.cc:523] T 37eda67598fe491bbb29aa62b2c4bec7 P da3584f429104d23a1a8899b35cc2138: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:42.036597 17499 consensus_queue.cc:237] T 49ef898a43704b87ba077cd7b0485e9a P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } }
I20240502 01:54:42.040026 19870 raft_consensus.cc:1261] T 49ef898a43704b87ba077cd7b0485e9a P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Refusing update from remote peer 311f765f59754bdcb4eadc04e14bd4ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:42.041663 17901 raft_consensus.cc:1261] T 49ef898a43704b87ba077cd7b0485e9a P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Refusing update from remote peer 311f765f59754bdcb4eadc04e14bd4ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:42.024206 19466 raft_consensus.cc:2932] T c91f74f46b04425aa5393515cb9f781c P 29806d3d37354c169484f6db368f8798 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
I20240502 01:54:42.042906 21938 consensus_queue.cc:1035] T 49ef898a43704b87ba077cd7b0485e9a P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:42.044078 21362 raft_consensus.cc:2932] T c91f74f46b04425aa5393515cb9f781c P 84d3a4b174c34e02aedab1e620731359 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
W20240502 01:54:42.045010 17432 consensus_peers.cc:485] T 49ef898a43704b87ba077cd7b0485e9a P 311f765f59754bdcb4eadc04e14bd4ea -> Peer 29806d3d37354c169484f6db368f8798 (127.9.176.197:35011): Couldn't send request to peer 29806d3d37354c169484f6db368f8798. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 49ef898a43704b87ba077cd7b0485e9a. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:42.046764 21938 consensus_queue.cc:1035] T 49ef898a43704b87ba077cd7b0485e9a P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:42.048920 21907 tablet_copy_client.cc:308] T 2045e08a200148c69326351f38fd484d P da3584f429104d23a1a8899b35cc2138: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.195:37357
I20240502 01:54:42.049172 17343 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c91f74f46b04425aa5393515cb9f781c with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:42.049796 17789 tablet_copy_service.cc:140] P 84d3a4b174c34e02aedab1e620731359: Received BeginTabletCopySession request for tablet 2045e08a200148c69326351f38fd484d from peer da3584f429104d23a1a8899b35cc2138 ({username='slave'} at 127.9.176.196:49859)
I20240502 01:54:42.050037 17789 tablet_copy_service.cc:161] P 84d3a4b174c34e02aedab1e620731359: Beginning new tablet copy session on tablet 2045e08a200148c69326351f38fd484d from peer da3584f429104d23a1a8899b35cc2138 at {username='slave'} at 127.9.176.196:49859: session id = da3584f429104d23a1a8899b35cc2138-2045e08a200148c69326351f38fd484d
I20240502 01:54:42.051348 21467 raft_consensus.cc:2932] T 49ef898a43704b87ba077cd7b0485e9a P 311f765f59754bdcb4eadc04e14bd4ea [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 1b50a031f2784ba498b3e52e16916205 (127.9.176.194) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } } }
I20240502 01:54:42.052495 19604 raft_consensus.cc:2932] T c91f74f46b04425aa5393515cb9f781c P 4b7c382c9edd41babdd1a1afc904a813 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
I20240502 01:54:42.054046 17347 catalog_manager.cc:5484] T c91f74f46b04425aa5393515cb9f781c P 84d3a4b174c34e02aedab1e620731359 reported cstate change: config changed from index 8 to 9, VOTER 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193) evicted. New cstate: current_term: 1 leader_uuid: "84d3a4b174c34e02aedab1e620731359" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:42.057507 17789 tablet_copy_source_session.cc:205] T 2045e08a200148c69326351f38fd484d P 84d3a4b174c34e02aedab1e620731359: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:42.057343 17901 raft_consensus.cc:2932] T 49ef898a43704b87ba077cd7b0485e9a P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 1b50a031f2784ba498b3e52e16916205 (127.9.176.194) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } } }
I20240502 01:54:42.059396 17478 tablet_service.cc:1514] Processing DeleteTablet for tablet c91f74f46b04425aa5393515cb9f781c with delete_type TABLET_DATA_TOMBSTONED (TS 311f765f59754bdcb4eadc04e14bd4ea not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:48974
I20240502 01:54:42.061584 19870 raft_consensus.cc:2932] T 49ef898a43704b87ba077cd7b0485e9a P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 1b50a031f2784ba498b3e52e16916205 (127.9.176.194) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } } }
I20240502 01:54:42.062217 21907 ts_tablet_manager.cc:1875] T 2045e08a200148c69326351f38fd484d P da3584f429104d23a1a8899b35cc2138: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 01:54:42.073294 21930 tablet_bootstrap.cc:492] T 37eda67598fe491bbb29aa62b2c4bec7 P da3584f429104d23a1a8899b35cc2138: Bootstrap starting.
I20240502 01:54:42.075132 17343 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 49ef898a43704b87ba077cd7b0485e9a with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:42.085904 21907 ts_tablet_manager.cc:1888] T 2045e08a200148c69326351f38fd484d P da3584f429104d23a1a8899b35cc2138: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240502 01:54:42.086189 17355 catalog_manager.cc:5484] T 49ef898a43704b87ba077cd7b0485e9a P 94931f4b51c9493cb3d4264ce99f5e76 reported cstate change: config changed from index 7 to 8, VOTER 1b50a031f2784ba498b3e52e16916205 (127.9.176.194) evicted. New cstate: current_term: 1 leader_uuid: "311f765f59754bdcb4eadc04e14bd4ea" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } } }
I20240502 01:54:42.087162 21907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2045e08a200148c69326351f38fd484d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:42.092414 17613 tablet_service.cc:1514] Processing DeleteTablet for tablet 49ef898a43704b87ba077cd7b0485e9a with delete_type TABLET_DATA_TOMBSTONED (TS 1b50a031f2784ba498b3e52e16916205 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:41484
I20240502 01:54:42.103555 21907 tablet_copy_client.cc:783] T 2045e08a200148c69326351f38fd484d P da3584f429104d23a1a8899b35cc2138: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:42.104015 21907 tablet_copy_client.cc:647] T 2045e08a200148c69326351f38fd484d P da3584f429104d23a1a8899b35cc2138: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:42.109714 21907 tablet_copy_client.cc:523] T 2045e08a200148c69326351f38fd484d P da3584f429104d23a1a8899b35cc2138: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:42.033119 21842 raft_consensus.cc:348] T 6da59c0626c64a5988e39a831c42c268 P 272b2179c1b4476090a89880df2a8baa [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } }
I20240502 01:54:42.111912 21842 raft_consensus.cc:729] T 6da59c0626c64a5988e39a831c42c268 P 272b2179c1b4476090a89880df2a8baa [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 272b2179c1b4476090a89880df2a8baa, State: Initialized, Role: LEARNER
I20240502 01:54:42.113837 21907 tablet_bootstrap.cc:492] T 2045e08a200148c69326351f38fd484d P da3584f429104d23a1a8899b35cc2138: Bootstrap starting.
I20240502 01:54:42.114265 21842 consensus_queue.cc:260] T 6da59c0626c64a5988e39a831c42c268 P 272b2179c1b4476090a89880df2a8baa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } }
I20240502 01:54:42.115326 21844 tablet_bootstrap.cc:492] T 2c78951695284b1aaab411c0a041c28a P 29806d3d37354c169484f6db368f8798: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:42.120554 21842 ts_tablet_manager.cc:1398] T 6da59c0626c64a5988e39a831c42c268 P 272b2179c1b4476090a89880df2a8baa: Time spent starting tablet: real 0.089s	user 0.002s	sys 0.000s
I20240502 01:54:42.124346 17789 tablet_copy_service.cc:342] P 84d3a4b174c34e02aedab1e620731359: Request end of tablet copy session 272b2179c1b4476090a89880df2a8baa-6da59c0626c64a5988e39a831c42c268 received from {username='slave'} at 127.9.176.199:46159
I20240502 01:54:42.124552 17789 tablet_copy_service.cc:433] P 84d3a4b174c34e02aedab1e620731359: ending tablet copy session 272b2179c1b4476090a89880df2a8baa-6da59c0626c64a5988e39a831c42c268 on tablet 6da59c0626c64a5988e39a831c42c268 with peer 272b2179c1b4476090a89880df2a8baa
I20240502 01:54:42.126504 21778 raft_consensus.cc:1050] T 05b67d9912f047bfb6a2bed90b82ae15 P da3584f429104d23a1a8899b35cc2138: attempting to promote NON_VOTER 94931f4b51c9493cb3d4264ce99f5e76 to VOTER
I20240502 01:54:42.130350 21956 tablet_replica.cc:330] T 49ef898a43704b87ba077cd7b0485e9a P 1b50a031f2784ba498b3e52e16916205: stopping tablet replica
I20240502 01:54:42.130671 21956 raft_consensus.cc:2229] T 49ef898a43704b87ba077cd7b0485e9a P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:42.130198 21778 consensus_queue.cc:237] T 05b67d9912f047bfb6a2bed90b82ae15 P da3584f429104d23a1a8899b35cc2138 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } }
I20240502 01:54:42.131053 21956 raft_consensus.cc:2258] T 49ef898a43704b87ba077cd7b0485e9a P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:42.139176 21951 tablet_replica.cc:330] T c91f74f46b04425aa5393515cb9f781c P 311f765f59754bdcb4eadc04e14bd4ea: stopping tablet replica
I20240502 01:54:42.143913 21798 ts_tablet_manager.cc:1398] T 3b3fa060e8d64911bec810ec07a358d2 P da3584f429104d23a1a8899b35cc2138: Time spent starting tablet: real 0.127s	user 0.004s	sys 0.000s
I20240502 01:54:42.139503 21951 raft_consensus.cc:2229] T c91f74f46b04425aa5393515cb9f781c P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:42.150336 17654 tablet_copy_service.cc:342] P 1b50a031f2784ba498b3e52e16916205: Request end of tablet copy session da3584f429104d23a1a8899b35cc2138-3b3fa060e8d64911bec810ec07a358d2 received from {username='slave'} at 127.9.176.196:38229
I20240502 01:54:42.150532 17654 tablet_copy_service.cc:433] P 1b50a031f2784ba498b3e52e16916205: ending tablet copy session da3584f429104d23a1a8899b35cc2138-3b3fa060e8d64911bec810ec07a358d2 on tablet 3b3fa060e8d64911bec810ec07a358d2 with peer da3584f429104d23a1a8899b35cc2138
I20240502 01:54:42.156984 19464 raft_consensus.cc:1261] T c91f74f46b04425aa5393515cb9f781c P 29806d3d37354c169484f6db368f8798 [term 1 FOLLOWER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:54:42.158069 21951 raft_consensus.cc:2258] T c91f74f46b04425aa5393515cb9f781c P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:42.158126 21956 ts_tablet_manager.cc:1875] T 49ef898a43704b87ba077cd7b0485e9a P 1b50a031f2784ba498b3e52e16916205: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:42.158953 21777 consensus_queue.cc:1035] T c91f74f46b04425aa5393515cb9f781c P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:42.165277 19464 raft_consensus.cc:2932] T c91f74f46b04425aa5393515cb9f781c P 29806d3d37354c169484f6db368f8798 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
I20240502 01:54:42.168720 19604 raft_consensus.cc:1261] T 05b67d9912f047bfb6a2bed90b82ae15 P 4b7c382c9edd41babdd1a1afc904a813 [term 1 FOLLOWER]: Refusing update from remote peer da3584f429104d23a1a8899b35cc2138: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:42.169051 17764 raft_consensus.cc:1261] T 05b67d9912f047bfb6a2bed90b82ae15 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Refusing update from remote peer da3584f429104d23a1a8899b35cc2138: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:42.169571 20915 consensus_queue.cc:1035] T 05b67d9912f047bfb6a2bed90b82ae15 P da3584f429104d23a1a8899b35cc2138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:42.170192 19870 raft_consensus.cc:1261] T 05b67d9912f047bfb6a2bed90b82ae15 P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 LEARNER]: Refusing update from remote peer da3584f429104d23a1a8899b35cc2138: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:42.170279 21695 consensus_queue.cc:1035] T 05b67d9912f047bfb6a2bed90b82ae15 P da3584f429104d23a1a8899b35cc2138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:42.170805 20915 consensus_queue.cc:1035] T 05b67d9912f047bfb6a2bed90b82ae15 P da3584f429104d23a1a8899b35cc2138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:42.172694 21844 tablet_bootstrap.cc:492] T 2c78951695284b1aaab411c0a041c28a P 29806d3d37354c169484f6db368f8798: Bootstrap complete.
I20240502 01:54:42.173074 21844 ts_tablet_manager.cc:1367] T 2c78951695284b1aaab411c0a041c28a P 29806d3d37354c169484f6db368f8798: Time spent bootstrapping tablet: real 0.544s	user 0.066s	sys 0.004s
I20240502 01:54:42.174010 21844 raft_consensus.cc:348] T 2c78951695284b1aaab411c0a041c28a P 29806d3d37354c169484f6db368f8798 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } }
I20240502 01:54:42.174386 21844 raft_consensus.cc:729] T 2c78951695284b1aaab411c0a041c28a P 29806d3d37354c169484f6db368f8798 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 29806d3d37354c169484f6db368f8798, State: Initialized, Role: LEARNER
I20240502 01:54:42.174706 21844 consensus_queue.cc:260] T 2c78951695284b1aaab411c0a041c28a P 29806d3d37354c169484f6db368f8798 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } }
I20240502 01:54:42.182796 21951 ts_tablet_manager.cc:1875] T c91f74f46b04425aa5393515cb9f781c P 311f765f59754bdcb4eadc04e14bd4ea: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:42.194459 17500 consensus_queue.cc:237] T 59e507855e5043d38cef84e92adf437b P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } }
I20240502 01:54:42.200168 20915 raft_consensus.cc:2932] T 05b67d9912f047bfb6a2bed90b82ae15 P da3584f429104d23a1a8899b35cc2138 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
I20240502 01:54:42.207927 21956 ts_tablet_manager.cc:1888] T 49ef898a43704b87ba077cd7b0485e9a P 1b50a031f2784ba498b3e52e16916205: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240502 01:54:42.208329 19870 raft_consensus.cc:2932] T 05b67d9912f047bfb6a2bed90b82ae15 P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
I20240502 01:54:42.211514 17901 raft_consensus.cc:1261] T 59e507855e5043d38cef84e92adf437b P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Refusing update from remote peer 311f765f59754bdcb4eadc04e14bd4ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:42.212369 21467 consensus_queue.cc:1035] T 59e507855e5043d38cef84e92adf437b P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:42.214951 17767 raft_consensus.cc:2932] T 05b67d9912f047bfb6a2bed90b82ae15 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
I20240502 01:54:42.215914 17347 catalog_manager.cc:5484] T 05b67d9912f047bfb6a2bed90b82ae15 P 94931f4b51c9493cb3d4264ce99f5e76 reported cstate change: config changed from index 7 to 8, 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "da3584f429104d23a1a8899b35cc2138" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
I20240502 01:54:42.221009 19736 raft_consensus.cc:1203] T 6da59c0626c64a5988e39a831c42c268 P 272b2179c1b4476090a89880df2a8baa [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10]   Dedup: 1.10->[]
I20240502 01:54:42.226362 17897 consensus_queue.cc:237] T 05b67d9912f047bfb6a2bed90b82ae15 P da3584f429104d23a1a8899b35cc2138 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } }
I20240502 01:54:42.234102 21956 log.cc:1196] T 49ef898a43704b87ba077cd7b0485e9a P 1b50a031f2784ba498b3e52e16916205: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/wal/wals/49ef898a43704b87ba077cd7b0485e9a
I20240502 01:54:42.234128 19870 raft_consensus.cc:1261] T 05b67d9912f047bfb6a2bed90b82ae15 P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Refusing update from remote peer da3584f429104d23a1a8899b35cc2138: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:54:42.234771 20915 consensus_queue.cc:1035] T 05b67d9912f047bfb6a2bed90b82ae15 P da3584f429104d23a1a8899b35cc2138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:42.224372 17631 raft_consensus.cc:1261] T 59e507855e5043d38cef84e92adf437b P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Refusing update from remote peer 311f765f59754bdcb4eadc04e14bd4ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:42.238984 20915 raft_consensus.cc:2932] T 05b67d9912f047bfb6a2bed90b82ae15 P da3584f429104d23a1a8899b35cc2138 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 84d3a4b174c34e02aedab1e620731359 (127.9.176.195) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
I20240502 01:54:42.240051 21467 consensus_queue.cc:1035] T 59e507855e5043d38cef84e92adf437b P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:42.240885 17343 catalog_manager.cc:4830] TS 1b50a031f2784ba498b3e52e16916205 (127.9.176.194:45617): tablet 49ef898a43704b87ba077cd7b0485e9a (table rebalance_test_table_8 [id=40a6be2c1a2744428cea56b78225e44e]) successfully deleted
I20240502 01:54:42.246798 19466 raft_consensus.cc:1203] T 2c78951695284b1aaab411c0a041c28a P 29806d3d37354c169484f6db368f8798 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11]   Dedup: 1.11->[]
W20240502 01:54:42.251161 17432 consensus_peers.cc:485] T 59e507855e5043d38cef84e92adf437b P 311f765f59754bdcb4eadc04e14bd4ea -> Peer 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198:36865): Couldn't send request to peer 4b7c382c9edd41babdd1a1afc904a813. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 59e507855e5043d38cef84e92adf437b. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:42.250851 19603 raft_consensus.cc:2932] T 05b67d9912f047bfb6a2bed90b82ae15 P 4b7c382c9edd41babdd1a1afc904a813 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
I20240502 01:54:42.252555 21844 ts_tablet_manager.cc:1398] T 2c78951695284b1aaab411c0a041c28a P 29806d3d37354c169484f6db368f8798: Time spent starting tablet: real 0.079s	user 0.003s	sys 0.000s
I20240502 01:54:42.253489 17654 tablet_copy_service.cc:342] P 1b50a031f2784ba498b3e52e16916205: Request end of tablet copy session 29806d3d37354c169484f6db368f8798-2c78951695284b1aaab411c0a041c28a received from {username='slave'} at 127.9.176.197:33725
I20240502 01:54:42.253660 17654 tablet_copy_service.cc:433] P 1b50a031f2784ba498b3e52e16916205: ending tablet copy session 29806d3d37354c169484f6db368f8798-2c78951695284b1aaab411c0a041c28a on tablet 2c78951695284b1aaab411c0a041c28a with peer 29806d3d37354c169484f6db368f8798
I20240502 01:54:42.257372 19604 raft_consensus.cc:1261] T 05b67d9912f047bfb6a2bed90b82ae15 P 4b7c382c9edd41babdd1a1afc904a813 [term 1 FOLLOWER]: Refusing update from remote peer da3584f429104d23a1a8899b35cc2138: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:54:42.273023 21778 consensus_queue.cc:1035] T 05b67d9912f047bfb6a2bed90b82ae15 P da3584f429104d23a1a8899b35cc2138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:42.280884 17340 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 05b67d9912f047bfb6a2bed90b82ae15 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:42.301342 17478 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:54:42.287590 21951 ts_tablet_manager.cc:1888] T c91f74f46b04425aa5393515cb9f781c P 311f765f59754bdcb4eadc04e14bd4ea: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 01:54:42.307816 21951 log.cc:1196] T c91f74f46b04425aa5393515cb9f781c P 311f765f59754bdcb4eadc04e14bd4ea: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/wal/wals/c91f74f46b04425aa5393515cb9f781c
I20240502 01:54:42.308984 17343 catalog_manager.cc:4830] TS 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193:35397): tablet c91f74f46b04425aa5393515cb9f781c (table rebalance_test_table_2 [id=bf67e64a43cd4367a7420c6bf1793faa]) successfully deleted
I20240502 01:54:42.320730 21409 raft_consensus.cc:2932] T 59e507855e5043d38cef84e92adf437b P 311f765f59754bdcb4eadc04e14bd4ea [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } } }
I20240502 01:54:42.321415 19870 raft_consensus.cc:2932] T 05b67d9912f047bfb6a2bed90b82ae15 P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 84d3a4b174c34e02aedab1e620731359 (127.9.176.195) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
I20240502 01:54:42.331398 19718 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:42.331672 17347 catalog_manager.cc:5484] T 59e507855e5043d38cef84e92adf437b P 311f765f59754bdcb4eadc04e14bd4ea reported cstate change: config changed from index -1 to 4, NON_VOTER 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) added. New cstate: current_term: 1 leader_uuid: "311f765f59754bdcb4eadc04e14bd4ea" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:42.336941 21981 ts_tablet_manager.cc:902] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 84d3a4b174c34e02aedab1e620731359: Initiating tablet copy from peer da3584f429104d23a1a8899b35cc2138 (127.9.176.196:46381)
I20240502 01:54:42.338115 17901 raft_consensus.cc:2932] T 59e507855e5043d38cef84e92adf437b P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } } }
I20240502 01:54:42.341131 17355 catalog_manager.cc:5484] T 05b67d9912f047bfb6a2bed90b82ae15 P da3584f429104d23a1a8899b35cc2138 reported cstate change: config changed from index 8 to 9, VOTER 84d3a4b174c34e02aedab1e620731359 (127.9.176.195) evicted. New cstate: current_term: 1 leader_uuid: "da3584f429104d23a1a8899b35cc2138" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:42.350560 17746 tablet_service.cc:1514] Processing DeleteTablet for tablet 05b67d9912f047bfb6a2bed90b82ae15 with delete_type TABLET_DATA_TOMBSTONED (TS 84d3a4b174c34e02aedab1e620731359 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:47166
W20240502 01:54:42.356493 17832 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.176.198:36865, user_credentials={real_user=slave}} blocked reactor thread for 98433.2us
I20240502 01:54:42.357072 17613 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:54:42.359387 19604 raft_consensus.cc:2932] T 05b67d9912f047bfb6a2bed90b82ae15 P 4b7c382c9edd41babdd1a1afc904a813 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 84d3a4b174c34e02aedab1e620731359 (127.9.176.195) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
I20240502 01:54:42.385581 17745 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:54:42.392082 21989 tablet_replica.cc:330] T 05b67d9912f047bfb6a2bed90b82ae15 P 84d3a4b174c34e02aedab1e620731359: stopping tablet replica
I20240502 01:54:42.392485 21989 raft_consensus.cc:2229] T 05b67d9912f047bfb6a2bed90b82ae15 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:42.393862 17634 raft_consensus.cc:2932] T 59e507855e5043d38cef84e92adf437b P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } } }
I20240502 01:54:42.402186 19585 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:42.403101 21981 tablet_copy_client.cc:308] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 84d3a4b174c34e02aedab1e620731359: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.196:46381
I20240502 01:54:42.404140 17922 tablet_copy_service.cc:140] P da3584f429104d23a1a8899b35cc2138: Received BeginTabletCopySession request for tablet 60ff2e50b1d74e0c8e0890b87f9ac8ba from peer 84d3a4b174c34e02aedab1e620731359 ({username='slave'} at 127.9.176.195:43669)
I20240502 01:54:42.404366 17922 tablet_copy_service.cc:161] P da3584f429104d23a1a8899b35cc2138: Beginning new tablet copy session on tablet 60ff2e50b1d74e0c8e0890b87f9ac8ba from peer 84d3a4b174c34e02aedab1e620731359 at {username='slave'} at 127.9.176.195:43669: session id = 84d3a4b174c34e02aedab1e620731359-60ff2e50b1d74e0c8e0890b87f9ac8ba
I20240502 01:54:42.408708 17922 tablet_copy_source_session.cc:205] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P da3584f429104d23a1a8899b35cc2138: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:42.411188 21981 ts_tablet_manager.cc:1875] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 84d3a4b174c34e02aedab1e620731359: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 01:54:42.420406 21989 raft_consensus.cc:2258] T 05b67d9912f047bfb6a2bed90b82ae15 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:42.422636 21989 ts_tablet_manager.cc:1875] T 05b67d9912f047bfb6a2bed90b82ae15 P 84d3a4b174c34e02aedab1e620731359: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:42.438844 19852 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:42.468246 19446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:42.488402 21981 ts_tablet_manager.cc:1888] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 84d3a4b174c34e02aedab1e620731359: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.8
I20240502 01:54:42.488952 21981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60ff2e50b1d74e0c8e0890b87f9ac8ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:42.510342 21777 raft_consensus.cc:1050] T f07e9b433e554c85b3c25b7e708f8b2f P 84d3a4b174c34e02aedab1e620731359: attempting to promote NON_VOTER 4b7c382c9edd41babdd1a1afc904a813 to VOTER
I20240502 01:54:42.511339 21777 consensus_queue.cc:237] T f07e9b433e554c85b3c25b7e708f8b2f P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } }
I20240502 01:54:42.519326 17880 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:54:42.523265 17901 raft_consensus.cc:1203] T 3b3fa060e8d64911bec810ec07a358d2 P da3584f429104d23a1a8899b35cc2138 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8]   Dedup: 1.8->[]
I20240502 01:54:42.544589 21981 tablet_copy_client.cc:783] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 84d3a4b174c34e02aedab1e620731359: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:42.545073 21981 tablet_copy_client.cc:647] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 84d3a4b174c34e02aedab1e620731359: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:42.548400 19870 raft_consensus.cc:1261] T f07e9b433e554c85b3c25b7e708f8b2f P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:42.549217 21362 consensus_queue.cc:1035] T f07e9b433e554c85b3c25b7e708f8b2f P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:42.552980 19604 raft_consensus.cc:1261] T f07e9b433e554c85b3c25b7e708f8b2f P 4b7c382c9edd41babdd1a1afc904a813 [term 1 LEARNER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:42.553951 21777 consensus_queue.cc:1035] T f07e9b433e554c85b3c25b7e708f8b2f P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:42.554473 17631 raft_consensus.cc:1261] T f07e9b433e554c85b3c25b7e708f8b2f P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:42.555347 21362 consensus_queue.cc:1035] T f07e9b433e554c85b3c25b7e708f8b2f P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:42.559607 21989 ts_tablet_manager.cc:1888] T 05b67d9912f047bfb6a2bed90b82ae15 P 84d3a4b174c34e02aedab1e620731359: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 01:54:42.562026 21989 log.cc:1196] T 05b67d9912f047bfb6a2bed90b82ae15 P 84d3a4b174c34e02aedab1e620731359: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/wal/wals/05b67d9912f047bfb6a2bed90b82ae15
I20240502 01:54:42.561501 21981 tablet_copy_client.cc:523] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 84d3a4b174c34e02aedab1e620731359: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:42.573585 17343 catalog_manager.cc:4830] TS 84d3a4b174c34e02aedab1e620731359 (127.9.176.195:37357): tablet 05b67d9912f047bfb6a2bed90b82ae15 (table rebalance_test_table_9 [id=60aa7e7a81564c3a888cc9b68700c9ad]) successfully deleted
I20240502 01:54:42.580850 21776 raft_consensus.cc:2932] T f07e9b433e554c85b3c25b7e708f8b2f P 84d3a4b174c34e02aedab1e620731359 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
I20240502 01:54:42.591441 19603 raft_consensus.cc:2932] T f07e9b433e554c85b3c25b7e708f8b2f P 4b7c382c9edd41babdd1a1afc904a813 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
I20240502 01:54:42.591396 19870 raft_consensus.cc:2932] T f07e9b433e554c85b3c25b7e708f8b2f P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
I20240502 01:54:42.592114 17347 catalog_manager.cc:5484] T f07e9b433e554c85b3c25b7e708f8b2f P 84d3a4b174c34e02aedab1e620731359 reported cstate change: config changed from index 7 to 8, 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "84d3a4b174c34e02aedab1e620731359" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:42.603085 21882 tablet_bootstrap.cc:492] T 3b3fa060e8d64911bec810ec07a358d2 P 4b7c382c9edd41babdd1a1afc904a813: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:42.604153 21882 tablet_bootstrap.cc:492] T 3b3fa060e8d64911bec810ec07a358d2 P 4b7c382c9edd41babdd1a1afc904a813: Bootstrap complete.
I20240502 01:54:42.613638 21844 ts_tablet_manager.cc:902] T 49ef898a43704b87ba077cd7b0485e9a P 29806d3d37354c169484f6db368f8798: Initiating tablet copy from peer 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193:35397)
I20240502 01:54:42.615232 17767 consensus_queue.cc:237] T f07e9b433e554c85b3c25b7e708f8b2f P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } }
I20240502 01:54:42.618032 19604 raft_consensus.cc:1261] T f07e9b433e554c85b3c25b7e708f8b2f P 4b7c382c9edd41babdd1a1afc904a813 [term 1 FOLLOWER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:54:42.618456 19870 raft_consensus.cc:1261] T f07e9b433e554c85b3c25b7e708f8b2f P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:54:42.618716 21777 consensus_queue.cc:1035] T f07e9b433e554c85b3c25b7e708f8b2f P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:42.619132 21362 consensus_queue.cc:1035] T f07e9b433e554c85b3c25b7e708f8b2f P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:42.622066 21362 raft_consensus.cc:2932] T f07e9b433e554c85b3c25b7e708f8b2f P 84d3a4b174c34e02aedab1e620731359 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 1b50a031f2784ba498b3e52e16916205 (127.9.176.194) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
I20240502 01:54:42.622709 19603 raft_consensus.cc:2932] T f07e9b433e554c85b3c25b7e708f8b2f P 4b7c382c9edd41babdd1a1afc904a813 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 1b50a031f2784ba498b3e52e16916205 (127.9.176.194) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
I20240502 01:54:42.627918 17343 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f07e9b433e554c85b3c25b7e708f8b2f with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:42.632273 19869 raft_consensus.cc:2932] T f07e9b433e554c85b3c25b7e708f8b2f P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 1b50a031f2784ba498b3e52e16916205 (127.9.176.194) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
I20240502 01:54:42.632941 17355 catalog_manager.cc:5484] T f07e9b433e554c85b3c25b7e708f8b2f P 4b7c382c9edd41babdd1a1afc904a813 reported cstate change: config changed from index 8 to 9, VOTER 1b50a031f2784ba498b3e52e16916205 (127.9.176.194) evicted. New cstate: current_term: 1 leader_uuid: "84d3a4b174c34e02aedab1e620731359" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
I20240502 01:54:42.638339 17613 tablet_service.cc:1514] Processing DeleteTablet for tablet f07e9b433e554c85b3c25b7e708f8b2f with delete_type TABLET_DATA_TOMBSTONED (TS 1b50a031f2784ba498b3e52e16916205 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:41484
I20240502 01:54:42.638743 21956 tablet_replica.cc:330] T f07e9b433e554c85b3c25b7e708f8b2f P 1b50a031f2784ba498b3e52e16916205: stopping tablet replica
I20240502 01:54:42.638955 21956 raft_consensus.cc:2229] T f07e9b433e554c85b3c25b7e708f8b2f P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:42.639165 21956 pending_rounds.cc:62] T f07e9b433e554c85b3c25b7e708f8b2f P 1b50a031f2784ba498b3e52e16916205: Trying to abort 1 pending ops.
I20240502 01:54:42.639294 21956 pending_rounds.cc:68] T f07e9b433e554c85b3c25b7e708f8b2f P 1b50a031f2784ba498b3e52e16916205: Aborting op as it isn't in flight: id { term: 1 index: 8 } timestamp: 7023062558763667456 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "f07e9b433e554c85b3c25b7e708f8b2f" old_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } } } new_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } } }
I20240502 01:54:42.640010 21956 raft_consensus.cc:2894] T f07e9b433e554c85b3c25b7e708f8b2f P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Aborting config change with OpId 1.8: Aborted: Op aborted
I20240502 01:54:42.640154 21956 raft_consensus.cc:2866] T f07e9b433e554c85b3c25b7e708f8b2f P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240502 01:54:42.640311 21956 raft_consensus.cc:2258] T f07e9b433e554c85b3c25b7e708f8b2f P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:42.647243 21882 ts_tablet_manager.cc:1367] T 3b3fa060e8d64911bec810ec07a358d2 P 4b7c382c9edd41babdd1a1afc904a813: Time spent bootstrapping tablet: real 0.688s	user 0.055s	sys 0.016s
I20240502 01:54:42.648255 21882 raft_consensus.cc:348] T 3b3fa060e8d64911bec810ec07a358d2 P 4b7c382c9edd41babdd1a1afc904a813 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { promote: true } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } }
I20240502 01:54:42.648607 21882 raft_consensus.cc:729] T 3b3fa060e8d64911bec810ec07a358d2 P 4b7c382c9edd41babdd1a1afc904a813 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4b7c382c9edd41babdd1a1afc904a813, State: Initialized, Role: LEARNER
I20240502 01:54:42.648851 21882 consensus_queue.cc:260] T 3b3fa060e8d64911bec810ec07a358d2 P 4b7c382c9edd41babdd1a1afc904a813 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { promote: true } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } }
I20240502 01:54:42.649539 21882 ts_tablet_manager.cc:1398] T 3b3fa060e8d64911bec810ec07a358d2 P 4b7c382c9edd41babdd1a1afc904a813: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.001s
I20240502 01:54:42.660070 21981 tablet_bootstrap.cc:492] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 84d3a4b174c34e02aedab1e620731359: Bootstrap starting.
I20240502 01:54:42.660943 21926 raft_consensus.cc:1050] T 2c78951695284b1aaab411c0a041c28a P 1b50a031f2784ba498b3e52e16916205: attempting to promote NON_VOTER 29806d3d37354c169484f6db368f8798 to VOTER
I20240502 01:54:42.661998 21926 consensus_queue.cc:237] T 2c78951695284b1aaab411c0a041c28a P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } }
I20240502 01:54:42.662647 21956 ts_tablet_manager.cc:1875] T f07e9b433e554c85b3c25b7e708f8b2f P 1b50a031f2784ba498b3e52e16916205: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:42.674170 17654 tablet_copy_service.cc:342] P 1b50a031f2784ba498b3e52e16916205: Request end of tablet copy session 4b7c382c9edd41babdd1a1afc904a813-3b3fa060e8d64911bec810ec07a358d2 received from {username='slave'} at 127.9.176.198:49387
I20240502 01:54:42.674382 17654 tablet_copy_service.cc:433] P 1b50a031f2784ba498b3e52e16916205: ending tablet copy session 4b7c382c9edd41babdd1a1afc904a813-3b3fa060e8d64911bec810ec07a358d2 on tablet 3b3fa060e8d64911bec810ec07a358d2 with peer 4b7c382c9edd41babdd1a1afc904a813
I20240502 01:54:42.682292 19466 raft_consensus.cc:1261] T 2c78951695284b1aaab411c0a041c28a P 29806d3d37354c169484f6db368f8798 [term 1 LEARNER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 01:54:42.682400 19869 raft_consensus.cc:1261] T 2c78951695284b1aaab411c0a041c28a P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 01:54:42.683265 20628 consensus_queue.cc:1035] T 2c78951695284b1aaab411c0a041c28a P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 01:54:42.684132 20628 consensus_queue.cc:1035] T 2c78951695284b1aaab411c0a041c28a P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 01:54:42.690961 19736 raft_consensus.cc:1261] T 2c78951695284b1aaab411c0a041c28a P 272b2179c1b4476090a89880df2a8baa [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 01:54:42.691740 21630 consensus_queue.cc:1035] T 2c78951695284b1aaab411c0a041c28a P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 01:54:42.692373 21956 ts_tablet_manager.cc:1888] T f07e9b433e554c85b3c25b7e708f8b2f P 1b50a031f2784ba498b3e52e16916205: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 01:54:42.693359 21956 log.cc:1196] T f07e9b433e554c85b3c25b7e708f8b2f P 1b50a031f2784ba498b3e52e16916205: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/wal/wals/f07e9b433e554c85b3c25b7e708f8b2f
I20240502 01:54:42.695173 17343 catalog_manager.cc:4830] TS 1b50a031f2784ba498b3e52e16916205 (127.9.176.194:45617): tablet f07e9b433e554c85b3c25b7e708f8b2f (table rebalance_test_table_4 [id=14b25b3f4ede4b65870815b014d90449]) successfully deleted
I20240502 01:54:42.702221 21882 ts_tablet_manager.cc:902] T 59e507855e5043d38cef84e92adf437b P 4b7c382c9edd41babdd1a1afc904a813: Initiating tablet copy from peer 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193:35397)
I20240502 01:54:42.715049 21630 raft_consensus.cc:2932] T 2c78951695284b1aaab411c0a041c28a P 1b50a031f2784ba498b3e52e16916205 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 29806d3d37354c169484f6db368f8798 (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } }
I20240502 01:54:42.719292 19736 raft_consensus.cc:2932] T 2c78951695284b1aaab411c0a041c28a P 272b2179c1b4476090a89880df2a8baa [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 29806d3d37354c169484f6db368f8798 (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } }
I20240502 01:54:42.727458 19869 raft_consensus.cc:2932] T 2c78951695284b1aaab411c0a041c28a P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 29806d3d37354c169484f6db368f8798 (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } }
I20240502 01:54:42.727839 17355 catalog_manager.cc:5484] T 2c78951695284b1aaab411c0a041c28a P 1b50a031f2784ba498b3e52e16916205 reported cstate change: config changed from index 11 to 12, 29806d3d37354c169484f6db368f8798 (127.9.176.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1b50a031f2784ba498b3e52e16916205" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:42.741922 21777 raft_consensus.cc:1050] T 6da59c0626c64a5988e39a831c42c268 P 84d3a4b174c34e02aedab1e620731359: attempting to promote NON_VOTER 272b2179c1b4476090a89880df2a8baa to VOTER
I20240502 01:54:42.742985 21777 consensus_queue.cc:237] T 6da59c0626c64a5988e39a831c42c268 P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } }
I20240502 01:54:42.768427 19464 raft_consensus.cc:2932] T 2c78951695284b1aaab411c0a041c28a P 29806d3d37354c169484f6db368f8798 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 29806d3d37354c169484f6db368f8798 (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } }
I20240502 01:54:42.788022 21844 tablet_copy_client.cc:308] T 49ef898a43704b87ba077cd7b0485e9a P 29806d3d37354c169484f6db368f8798: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.193:35397
I20240502 01:54:42.790037 17520 tablet_copy_service.cc:140] P 311f765f59754bdcb4eadc04e14bd4ea: Received BeginTabletCopySession request for tablet 49ef898a43704b87ba077cd7b0485e9a from peer 29806d3d37354c169484f6db368f8798 ({username='slave'} at 127.9.176.197:51011)
I20240502 01:54:42.790321 17520 tablet_copy_service.cc:161] P 311f765f59754bdcb4eadc04e14bd4ea: Beginning new tablet copy session on tablet 49ef898a43704b87ba077cd7b0485e9a from peer 29806d3d37354c169484f6db368f8798 at {username='slave'} at 127.9.176.197:51011: session id = 29806d3d37354c169484f6db368f8798-49ef898a43704b87ba077cd7b0485e9a
I20240502 01:54:42.794034 17520 tablet_copy_source_session.cc:205] T 49ef898a43704b87ba077cd7b0485e9a P 311f765f59754bdcb4eadc04e14bd4ea: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:42.795092 19736 raft_consensus.cc:1261] T 6da59c0626c64a5988e39a831c42c268 P 272b2179c1b4476090a89880df2a8baa [term 1 LEARNER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 01:54:42.795092 19869 raft_consensus.cc:1261] T 6da59c0626c64a5988e39a831c42c268 P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 01:54:42.795853 21362 consensus_queue.cc:1035] T 6da59c0626c64a5988e39a831c42c268 P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 01:54:42.797227 21362 consensus_queue.cc:1035] T 6da59c0626c64a5988e39a831c42c268 P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 01:54:42.797920 19466 raft_consensus.cc:1261] T 6da59c0626c64a5988e39a831c42c268 P 29806d3d37354c169484f6db368f8798 [term 1 FOLLOWER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 01:54:42.798578 21362 consensus_queue.cc:1035] T 6da59c0626c64a5988e39a831c42c268 P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 01:54:42.813066 21844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49ef898a43704b87ba077cd7b0485e9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:42.825773 21882 tablet_copy_client.cc:308] T 59e507855e5043d38cef84e92adf437b P 4b7c382c9edd41babdd1a1afc904a813: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.193:35397
I20240502 01:54:42.826948 17520 tablet_copy_service.cc:140] P 311f765f59754bdcb4eadc04e14bd4ea: Received BeginTabletCopySession request for tablet 59e507855e5043d38cef84e92adf437b from peer 4b7c382c9edd41babdd1a1afc904a813 ({username='slave'} at 127.9.176.198:53399)
I20240502 01:54:42.827148 17520 tablet_copy_service.cc:161] P 311f765f59754bdcb4eadc04e14bd4ea: Beginning new tablet copy session on tablet 59e507855e5043d38cef84e92adf437b from peer 4b7c382c9edd41babdd1a1afc904a813 at {username='slave'} at 127.9.176.198:53399: session id = 4b7c382c9edd41babdd1a1afc904a813-59e507855e5043d38cef84e92adf437b
I20240502 01:54:42.829648 21777 raft_consensus.cc:2932] T 6da59c0626c64a5988e39a831c42c268 P 84d3a4b174c34e02aedab1e620731359 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 272b2179c1b4476090a89880df2a8baa (127.9.176.199) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } }
I20240502 01:54:42.831208 17520 tablet_copy_source_session.cc:205] T 59e507855e5043d38cef84e92adf437b P 311f765f59754bdcb4eadc04e14bd4ea: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:42.832933 21882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59e507855e5043d38cef84e92adf437b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:42.832597 19736 raft_consensus.cc:2932] T 6da59c0626c64a5988e39a831c42c268 P 272b2179c1b4476090a89880df2a8baa [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 272b2179c1b4476090a89880df2a8baa (127.9.176.199) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } }
I20240502 01:54:42.835896 19869 raft_consensus.cc:2932] T 6da59c0626c64a5988e39a831c42c268 P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 272b2179c1b4476090a89880df2a8baa (127.9.176.199) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } }
I20240502 01:54:42.838469 17355 catalog_manager.cc:5484] T 6da59c0626c64a5988e39a831c42c268 P 84d3a4b174c34e02aedab1e620731359 reported cstate change: config changed from index 10 to 11, 272b2179c1b4476090a89880df2a8baa (127.9.176.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "84d3a4b174c34e02aedab1e620731359" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:42.862519 21930 tablet_bootstrap.cc:492] T 37eda67598fe491bbb29aa62b2c4bec7 P da3584f429104d23a1a8899b35cc2138: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:42.881049 21844 tablet_copy_client.cc:783] T 49ef898a43704b87ba077cd7b0485e9a P 29806d3d37354c169484f6db368f8798: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:42.882668 21882 tablet_copy_client.cc:783] T 59e507855e5043d38cef84e92adf437b P 4b7c382c9edd41babdd1a1afc904a813: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:42.888638 21930 tablet_bootstrap.cc:492] T 37eda67598fe491bbb29aa62b2c4bec7 P da3584f429104d23a1a8899b35cc2138: Bootstrap complete.
I20240502 01:54:42.897714 21844 tablet_copy_client.cc:647] T 49ef898a43704b87ba077cd7b0485e9a P 29806d3d37354c169484f6db368f8798: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:42.900012 21844 tablet_copy_client.cc:523] T 49ef898a43704b87ba077cd7b0485e9a P 29806d3d37354c169484f6db368f8798: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:42.907807 19464 raft_consensus.cc:2932] T 6da59c0626c64a5988e39a831c42c268 P 29806d3d37354c169484f6db368f8798 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 272b2179c1b4476090a89880df2a8baa (127.9.176.199) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } }
I20240502 01:54:42.909265 21930 ts_tablet_manager.cc:1367] T 37eda67598fe491bbb29aa62b2c4bec7 P da3584f429104d23a1a8899b35cc2138: Time spent bootstrapping tablet: real 0.836s	user 0.071s	sys 0.003s
I20240502 01:54:42.910202 21930 raft_consensus.cc:348] T 37eda67598fe491bbb29aa62b2c4bec7 P da3584f429104d23a1a8899b35cc2138 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { promote: true } }
I20240502 01:54:42.910501 21930 raft_consensus.cc:729] T 37eda67598fe491bbb29aa62b2c4bec7 P da3584f429104d23a1a8899b35cc2138 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: da3584f429104d23a1a8899b35cc2138, State: Initialized, Role: LEARNER
I20240502 01:54:42.910728 21930 consensus_queue.cc:260] T 37eda67598fe491bbb29aa62b2c4bec7 P da3584f429104d23a1a8899b35cc2138 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { promote: true } }
I20240502 01:54:42.915103 21882 tablet_copy_client.cc:647] T 59e507855e5043d38cef84e92adf437b P 4b7c382c9edd41babdd1a1afc904a813: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:42.915220 21930 ts_tablet_manager.cc:1398] T 37eda67598fe491bbb29aa62b2c4bec7 P da3584f429104d23a1a8899b35cc2138: Time spent starting tablet: real 0.006s	user 0.002s	sys 0.000s
I20240502 01:54:42.916005 17520 tablet_copy_service.cc:342] P 311f765f59754bdcb4eadc04e14bd4ea: Request end of tablet copy session da3584f429104d23a1a8899b35cc2138-37eda67598fe491bbb29aa62b2c4bec7 received from {username='slave'} at 127.9.176.196:38671
I20240502 01:54:42.916167 17520 tablet_copy_service.cc:433] P 311f765f59754bdcb4eadc04e14bd4ea: ending tablet copy session da3584f429104d23a1a8899b35cc2138-37eda67598fe491bbb29aa62b2c4bec7 on tablet 37eda67598fe491bbb29aa62b2c4bec7 with peer da3584f429104d23a1a8899b35cc2138
I20240502 01:54:42.920917 21882 tablet_copy_client.cc:523] T 59e507855e5043d38cef84e92adf437b P 4b7c382c9edd41babdd1a1afc904a813: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:42.924782 21844 tablet_bootstrap.cc:492] T 49ef898a43704b87ba077cd7b0485e9a P 29806d3d37354c169484f6db368f8798: Bootstrap starting.
I20240502 01:54:42.927356 21882 tablet_bootstrap.cc:492] T 59e507855e5043d38cef84e92adf437b P 4b7c382c9edd41babdd1a1afc904a813: Bootstrap starting.
I20240502 01:54:42.929555 21907 tablet_bootstrap.cc:492] T 2045e08a200148c69326351f38fd484d P da3584f429104d23a1a8899b35cc2138: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:42.930145 21907 tablet_bootstrap.cc:492] T 2045e08a200148c69326351f38fd484d P da3584f429104d23a1a8899b35cc2138: Bootstrap complete.
I20240502 01:54:42.934511 21907 ts_tablet_manager.cc:1367] T 2045e08a200148c69326351f38fd484d P da3584f429104d23a1a8899b35cc2138: Time spent bootstrapping tablet: real 0.821s	user 0.054s	sys 0.014s
I20240502 01:54:42.935606 21907 raft_consensus.cc:348] T 2045e08a200148c69326351f38fd484d P da3584f429104d23a1a8899b35cc2138 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { promote: true } }
I20240502 01:54:42.936025 21907 raft_consensus.cc:729] T 2045e08a200148c69326351f38fd484d P da3584f429104d23a1a8899b35cc2138 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: da3584f429104d23a1a8899b35cc2138, State: Initialized, Role: LEARNER
I20240502 01:54:42.936301 21907 consensus_queue.cc:260] T 2045e08a200148c69326351f38fd484d P da3584f429104d23a1a8899b35cc2138 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { promote: true } }
I20240502 01:54:42.936988 21907 ts_tablet_manager.cc:1398] T 2045e08a200148c69326351f38fd484d P da3584f429104d23a1a8899b35cc2138: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240502 01:54:42.940022 17789 tablet_copy_service.cc:342] P 84d3a4b174c34e02aedab1e620731359: Request end of tablet copy session da3584f429104d23a1a8899b35cc2138-2045e08a200148c69326351f38fd484d received from {username='slave'} at 127.9.176.196:49859
I20240502 01:54:42.940189 17789 tablet_copy_service.cc:433] P 84d3a4b174c34e02aedab1e620731359: ending tablet copy session da3584f429104d23a1a8899b35cc2138-2045e08a200148c69326351f38fd484d on tablet 2045e08a200148c69326351f38fd484d with peer da3584f429104d23a1a8899b35cc2138
I20240502 01:54:42.961488 19603 raft_consensus.cc:1203] T 3b3fa060e8d64911bec810ec07a358d2 P 4b7c382c9edd41babdd1a1afc904a813 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8]   Dedup: 1.8->[]
I20240502 01:54:43.034731 21630 raft_consensus.cc:1050] T 3b3fa060e8d64911bec810ec07a358d2 P 1b50a031f2784ba498b3e52e16916205: attempting to promote NON_VOTER da3584f429104d23a1a8899b35cc2138 to VOTER
I20240502 01:54:43.036108 21630 consensus_queue.cc:237] T 3b3fa060e8d64911bec810ec07a358d2 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } }
I20240502 01:54:43.046536 19603 raft_consensus.cc:1261] T 3b3fa060e8d64911bec810ec07a358d2 P 4b7c382c9edd41babdd1a1afc904a813 [term 1 LEARNER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:54:43.047264 19736 raft_consensus.cc:1261] T 3b3fa060e8d64911bec810ec07a358d2 P 272b2179c1b4476090a89880df2a8baa [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:54:43.048475 17901 raft_consensus.cc:1261] T 3b3fa060e8d64911bec810ec07a358d2 P da3584f429104d23a1a8899b35cc2138 [term 1 LEARNER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:54:43.050493 20628 consensus_queue.cc:1035] T 3b3fa060e8d64911bec810ec07a358d2 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:43.051138 20628 consensus_queue.cc:1035] T 3b3fa060e8d64911bec810ec07a358d2 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:43.055104 20628 consensus_queue.cc:1035] T 3b3fa060e8d64911bec810ec07a358d2 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:43.065274 17500 raft_consensus.cc:1261] T 3b3fa060e8d64911bec810ec07a358d2 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:54:43.067543 21926 consensus_queue.cc:1035] T 3b3fa060e8d64911bec810ec07a358d2 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:43.103642 21882 tablet_bootstrap.cc:492] T 59e507855e5043d38cef84e92adf437b P 4b7c382c9edd41babdd1a1afc904a813: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:43.105396 21882 tablet_bootstrap.cc:492] T 59e507855e5043d38cef84e92adf437b P 4b7c382c9edd41babdd1a1afc904a813: Bootstrap complete.
I20240502 01:54:43.105706 21882 ts_tablet_manager.cc:1367] T 59e507855e5043d38cef84e92adf437b P 4b7c382c9edd41babdd1a1afc904a813: Time spent bootstrapping tablet: real 0.178s	user 0.054s	sys 0.013s
I20240502 01:54:43.106731 21882 raft_consensus.cc:348] T 59e507855e5043d38cef84e92adf437b P 4b7c382c9edd41babdd1a1afc904a813 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } }
I20240502 01:54:43.107124 21882 raft_consensus.cc:729] T 59e507855e5043d38cef84e92adf437b P 4b7c382c9edd41babdd1a1afc904a813 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4b7c382c9edd41babdd1a1afc904a813, State: Initialized, Role: LEARNER
I20240502 01:54:43.107440 21882 consensus_queue.cc:260] T 59e507855e5043d38cef84e92adf437b P 4b7c382c9edd41babdd1a1afc904a813 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } }
I20240502 01:54:43.108143 21882 ts_tablet_manager.cc:1398] T 59e507855e5043d38cef84e92adf437b P 4b7c382c9edd41babdd1a1afc904a813: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.000s
I20240502 01:54:43.109145 17520 tablet_copy_service.cc:342] P 311f765f59754bdcb4eadc04e14bd4ea: Request end of tablet copy session 4b7c382c9edd41babdd1a1afc904a813-59e507855e5043d38cef84e92adf437b received from {username='slave'} at 127.9.176.198:53399
I20240502 01:54:43.109336 17520 tablet_copy_service.cc:433] P 311f765f59754bdcb4eadc04e14bd4ea: ending tablet copy session 4b7c382c9edd41babdd1a1afc904a813-59e507855e5043d38cef84e92adf437b on tablet 59e507855e5043d38cef84e92adf437b with peer 4b7c382c9edd41babdd1a1afc904a813
I20240502 01:54:43.132115 21926 raft_consensus.cc:1013] T 3b3fa060e8d64911bec810ec07a358d2 P 1b50a031f2784ba498b3e52e16916205 [term 1 LEADER]: attempt to promote peer 4b7c382c9edd41babdd1a1afc904a813: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 01:54:43.133124 21630 raft_consensus.cc:2932] T 3b3fa060e8d64911bec810ec07a358d2 P 1b50a031f2784ba498b3e52e16916205 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, da3584f429104d23a1a8899b35cc2138 (127.9.176.196) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } } }
I20240502 01:54:43.140403 19604 raft_consensus.cc:2932] T 3b3fa060e8d64911bec810ec07a358d2 P 4b7c382c9edd41babdd1a1afc904a813 [term 1 LEARNER]: Committing config change with OpId 1.9: config changed from index 8 to 9, da3584f429104d23a1a8899b35cc2138 (127.9.176.196) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } } }
I20240502 01:54:43.147262 17347 catalog_manager.cc:5484] T 3b3fa060e8d64911bec810ec07a358d2 P 1b50a031f2784ba498b3e52e16916205 reported cstate change: config changed from index 8 to 9, da3584f429104d23a1a8899b35cc2138 (127.9.176.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1b50a031f2784ba498b3e52e16916205" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } health_report { overall_health: HEALTHY } } }
I20240502 01:54:43.161141 21926 raft_consensus.cc:1050] T 3b3fa060e8d64911bec810ec07a358d2 P 1b50a031f2784ba498b3e52e16916205: attempting to promote NON_VOTER 4b7c382c9edd41babdd1a1afc904a813 to VOTER
I20240502 01:54:43.162092 21926 consensus_queue.cc:237] T 3b3fa060e8d64911bec810ec07a358d2 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } }
I20240502 01:54:43.166633 19604 raft_consensus.cc:1261] T 3b3fa060e8d64911bec810ec07a358d2 P 4b7c382c9edd41babdd1a1afc904a813 [term 1 LEARNER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:54:43.167484 21630 consensus_queue.cc:1035] T 3b3fa060e8d64911bec810ec07a358d2 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:54:43.172848 17499 raft_consensus.cc:1261] T 3b3fa060e8d64911bec810ec07a358d2 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:54:43.172847 19738 raft_consensus.cc:1261] T 3b3fa060e8d64911bec810ec07a358d2 P 272b2179c1b4476090a89880df2a8baa [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:54:43.173718 20628 consensus_queue.cc:1035] T 3b3fa060e8d64911bec810ec07a358d2 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:43.175458 19738 raft_consensus.cc:2932] T 3b3fa060e8d64911bec810ec07a358d2 P 272b2179c1b4476090a89880df2a8baa [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, da3584f429104d23a1a8899b35cc2138 (127.9.176.196) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } } }
I20240502 01:54:43.180325 21926 consensus_queue.cc:1035] T 3b3fa060e8d64911bec810ec07a358d2 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:43.186223 17500 raft_consensus.cc:2932] T 3b3fa060e8d64911bec810ec07a358d2 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, da3584f429104d23a1a8899b35cc2138 (127.9.176.196) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } } }
I20240502 01:54:43.189177 21926 raft_consensus.cc:2932] T 3b3fa060e8d64911bec810ec07a358d2 P 1b50a031f2784ba498b3e52e16916205 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
I20240502 01:54:43.193159 19738 raft_consensus.cc:2932] T 3b3fa060e8d64911bec810ec07a358d2 P 272b2179c1b4476090a89880df2a8baa [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
I20240502 01:54:43.208494 17898 raft_consensus.cc:1261] T 3b3fa060e8d64911bec810ec07a358d2 P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:54:43.211011 22086 consensus_queue.cc:1035] T 3b3fa060e8d64911bec810ec07a358d2 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:43.212714 17898 raft_consensus.cc:2932] T 3b3fa060e8d64911bec810ec07a358d2 P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, da3584f429104d23a1a8899b35cc2138 (127.9.176.196) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } } }
I20240502 01:54:43.217579 17898 raft_consensus.cc:2932] T 3b3fa060e8d64911bec810ec07a358d2 P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
I20240502 01:54:43.225528 19603 raft_consensus.cc:2932] T 3b3fa060e8d64911bec810ec07a358d2 P 4b7c382c9edd41babdd1a1afc904a813 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
I20240502 01:54:43.230392 17500 raft_consensus.cc:2932] T 3b3fa060e8d64911bec810ec07a358d2 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
I20240502 01:54:43.239759 19604 raft_consensus.cc:1203] T 59e507855e5043d38cef84e92adf437b P 4b7c382c9edd41babdd1a1afc904a813 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 01:54:43.231521 17355 catalog_manager.cc:5484] T 3b3fa060e8d64911bec810ec07a358d2 P 272b2179c1b4476090a89880df2a8baa reported cstate change: config changed from index 9 to 10, 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1b50a031f2784ba498b3e52e16916205" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
I20240502 01:54:43.274695 17500 consensus_queue.cc:237] T a6b5231a18a5414dbd010d4d48f08dbd P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } }
I20240502 01:54:43.298979 17898 raft_consensus.cc:1203] T 2045e08a200148c69326351f38fd484d P da3584f429104d23a1a8899b35cc2138 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240502 01:54:43.305286 17634 raft_consensus.cc:1261] T a6b5231a18a5414dbd010d4d48f08dbd P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Refusing update from remote peer 311f765f59754bdcb4eadc04e14bd4ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:43.306288 21938 consensus_queue.cc:1035] T a6b5231a18a5414dbd010d4d48f08dbd P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 01:54:43.307581 17432 consensus_peers.cc:485] T a6b5231a18a5414dbd010d4d48f08dbd P 311f765f59754bdcb4eadc04e14bd4ea -> Peer 29806d3d37354c169484f6db368f8798 (127.9.176.197:35011): Couldn't send request to peer 29806d3d37354c169484f6db368f8798. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a6b5231a18a5414dbd010d4d48f08dbd. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:43.317291 19603 raft_consensus.cc:1261] T a6b5231a18a5414dbd010d4d48f08dbd P 4b7c382c9edd41babdd1a1afc904a813 [term 1 FOLLOWER]: Refusing update from remote peer 311f765f59754bdcb4eadc04e14bd4ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:43.318118 21938 consensus_queue.cc:1035] T a6b5231a18a5414dbd010d4d48f08dbd P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:43.322131 17898 raft_consensus.cc:1203] T 37eda67598fe491bbb29aa62b2c4bec7 P da3584f429104d23a1a8899b35cc2138 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.11->[1.12-1.12]   Dedup: 1.12->[]
I20240502 01:54:43.335659 21938 raft_consensus.cc:2932] T a6b5231a18a5414dbd010d4d48f08dbd P 311f765f59754bdcb4eadc04e14bd4ea [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 29806d3d37354c169484f6db368f8798 (127.9.176.197) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } } }
I20240502 01:54:43.338919 17634 consensus_queue.cc:237] T 3b3fa060e8d64911bec810ec07a358d2 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } }
I20240502 01:54:43.342033 17898 raft_consensus.cc:1261] T 3b3fa060e8d64911bec810ec07a358d2 P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 01:54:43.342902 21926 consensus_queue.cc:1035] T 3b3fa060e8d64911bec810ec07a358d2 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 01:54:43.345917 19736 raft_consensus.cc:1261] T 3b3fa060e8d64911bec810ec07a358d2 P 272b2179c1b4476090a89880df2a8baa [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 01:54:43.347383 19603 raft_consensus.cc:1261] T 3b3fa060e8d64911bec810ec07a358d2 P 4b7c382c9edd41babdd1a1afc904a813 [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 01:54:43.348968 21926 consensus_queue.cc:1035] T 3b3fa060e8d64911bec810ec07a358d2 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 01:54:43.349563 21926 consensus_queue.cc:1035] T 3b3fa060e8d64911bec810ec07a358d2 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 01:54:43.350438 17634 raft_consensus.cc:2932] T a6b5231a18a5414dbd010d4d48f08dbd P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 29806d3d37354c169484f6db368f8798 (127.9.176.197) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } } }
I20240502 01:54:43.353103 19604 raft_consensus.cc:2932] T a6b5231a18a5414dbd010d4d48f08dbd P 4b7c382c9edd41babdd1a1afc904a813 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 29806d3d37354c169484f6db368f8798 (127.9.176.197) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } } }
I20240502 01:54:43.353104 22086 raft_consensus.cc:2932] T 3b3fa060e8d64911bec810ec07a358d2 P 1b50a031f2784ba498b3e52e16916205 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
I20240502 01:54:43.358438 17343 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3b3fa060e8d64911bec810ec07a358d2 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:43.359696 17347 catalog_manager.cc:5484] T a6b5231a18a5414dbd010d4d48f08dbd P 311f765f59754bdcb4eadc04e14bd4ea reported cstate change: config changed from index 6 to 7, NON_VOTER 29806d3d37354c169484f6db368f8798 (127.9.176.197) added. New cstate: current_term: 1 leader_uuid: "311f765f59754bdcb4eadc04e14bd4ea" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:43.363088 19736 raft_consensus.cc:2932] T 3b3fa060e8d64911bec810ec07a358d2 P 272b2179c1b4476090a89880df2a8baa [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
I20240502 01:54:43.363237 17898 raft_consensus.cc:2932] T 3b3fa060e8d64911bec810ec07a358d2 P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
I20240502 01:54:43.363678 17351 catalog_manager.cc:5484] T 3b3fa060e8d64911bec810ec07a358d2 P 1b50a031f2784ba498b3e52e16916205 reported cstate change: config changed from index 10 to 11, VOTER 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193) evicted. New cstate: current_term: 1 leader_uuid: "1b50a031f2784ba498b3e52e16916205" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:43.369984 17478 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b3fa060e8d64911bec810ec07a358d2 with delete_type TABLET_DATA_TOMBSTONED (TS 311f765f59754bdcb4eadc04e14bd4ea not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:48974
I20240502 01:54:43.385479 19602 raft_consensus.cc:2932] T 3b3fa060e8d64911bec810ec07a358d2 P 4b7c382c9edd41babdd1a1afc904a813 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
I20240502 01:54:43.402495 22108 tablet_replica.cc:330] T 3b3fa060e8d64911bec810ec07a358d2 P 311f765f59754bdcb4eadc04e14bd4ea: stopping tablet replica
I20240502 01:54:43.402905 22108 raft_consensus.cc:2229] T 3b3fa060e8d64911bec810ec07a358d2 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:43.403491 22108 raft_consensus.cc:2258] T 3b3fa060e8d64911bec810ec07a358d2 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:43.416054 21981 tablet_bootstrap.cc:492] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 84d3a4b174c34e02aedab1e620731359: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:43.420506 22108 ts_tablet_manager.cc:1875] T 3b3fa060e8d64911bec810ec07a358d2 P 311f765f59754bdcb4eadc04e14bd4ea: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:43.428207 21981 tablet_bootstrap.cc:492] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 84d3a4b174c34e02aedab1e620731359: Bootstrap complete.
I20240502 01:54:43.447306 21981 ts_tablet_manager.cc:1367] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 84d3a4b174c34e02aedab1e620731359: Time spent bootstrapping tablet: real 0.787s	user 0.064s	sys 0.003s
I20240502 01:54:43.443496 17767 consensus_queue.cc:237] T 88a137bcfc76410583f910b377d1be7f P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } }
I20240502 01:54:43.449040 21981 raft_consensus.cc:348] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 84d3a4b174c34e02aedab1e620731359 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: NON_VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { promote: true } }
I20240502 01:54:43.454257 21981 raft_consensus.cc:729] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 84d3a4b174c34e02aedab1e620731359 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 84d3a4b174c34e02aedab1e620731359, State: Initialized, Role: LEARNER
I20240502 01:54:43.454587 21981 consensus_queue.cc:260] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 84d3a4b174c34e02aedab1e620731359 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: NON_VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { promote: true } }
I20240502 01:54:43.460263 21981 ts_tablet_manager.cc:1398] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 84d3a4b174c34e02aedab1e620731359: Time spent starting tablet: real 0.012s	user 0.002s	sys 0.000s
I20240502 01:54:43.461081 17922 tablet_copy_service.cc:342] P da3584f429104d23a1a8899b35cc2138: Request end of tablet copy session 84d3a4b174c34e02aedab1e620731359-60ff2e50b1d74e0c8e0890b87f9ac8ba received from {username='slave'} at 127.9.176.195:43669
I20240502 01:54:43.461238 17922 tablet_copy_service.cc:433] P da3584f429104d23a1a8899b35cc2138: ending tablet copy session 84d3a4b174c34e02aedab1e620731359-60ff2e50b1d74e0c8e0890b87f9ac8ba on tablet 60ff2e50b1d74e0c8e0890b87f9ac8ba with peer 84d3a4b174c34e02aedab1e620731359
I20240502 01:54:43.462838 22108 ts_tablet_manager.cc:1888] T 3b3fa060e8d64911bec810ec07a358d2 P 311f765f59754bdcb4eadc04e14bd4ea: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.10
I20240502 01:54:43.464769 22108 log.cc:1196] T 3b3fa060e8d64911bec810ec07a358d2 P 311f765f59754bdcb4eadc04e14bd4ea: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/wal/wals/3b3fa060e8d64911bec810ec07a358d2
I20240502 01:54:43.469985 17343 catalog_manager.cc:4830] TS 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193:35397): tablet 3b3fa060e8d64911bec810ec07a358d2 (table rebalance_test_table_2 [id=bf67e64a43cd4367a7420c6bf1793faa]) successfully deleted
I20240502 01:54:43.483068 19604 raft_consensus.cc:1261] T 88a137bcfc76410583f910b377d1be7f P 4b7c382c9edd41babdd1a1afc904a813 [term 1 FOLLOWER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:43.484184 17500 raft_consensus.cc:1261] T 88a137bcfc76410583f910b377d1be7f P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:43.485041 21362 consensus_queue.cc:1035] T 88a137bcfc76410583f910b377d1be7f P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:43.485735 21362 consensus_queue.cc:1035] T 88a137bcfc76410583f910b377d1be7f P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 01:54:43.488993 17700 consensus_peers.cc:485] T 88a137bcfc76410583f910b377d1be7f P 84d3a4b174c34e02aedab1e620731359 -> Peer 272b2179c1b4476090a89880df2a8baa (127.9.176.199:42269): Couldn't send request to peer 272b2179c1b4476090a89880df2a8baa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 88a137bcfc76410583f910b377d1be7f. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:43.546411 17613 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:54:43.564275 21844 tablet_bootstrap.cc:492] T 49ef898a43704b87ba077cd7b0485e9a P 29806d3d37354c169484f6db368f8798: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:43.564858 21844 tablet_bootstrap.cc:492] T 49ef898a43704b87ba077cd7b0485e9a P 29806d3d37354c169484f6db368f8798: Bootstrap complete.
I20240502 01:54:43.569608 21844 ts_tablet_manager.cc:1367] T 49ef898a43704b87ba077cd7b0485e9a P 29806d3d37354c169484f6db368f8798: Time spent bootstrapping tablet: real 0.645s	user 0.047s	sys 0.021s
I20240502 01:54:43.572849 21844 raft_consensus.cc:348] T 49ef898a43704b87ba077cd7b0485e9a P 29806d3d37354c169484f6db368f8798 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } }
I20240502 01:54:43.573225 21844 raft_consensus.cc:729] T 49ef898a43704b87ba077cd7b0485e9a P 29806d3d37354c169484f6db368f8798 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 29806d3d37354c169484f6db368f8798, State: Initialized, Role: LEARNER
I20240502 01:54:43.573535 21844 consensus_queue.cc:260] T 49ef898a43704b87ba077cd7b0485e9a P 29806d3d37354c169484f6db368f8798 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } }
I20240502 01:54:43.578058 21777 raft_consensus.cc:2932] T 88a137bcfc76410583f910b377d1be7f P 84d3a4b174c34e02aedab1e620731359 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 272b2179c1b4476090a89880df2a8baa (127.9.176.199) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } } }
I20240502 01:54:43.580946 17499 raft_consensus.cc:2932] T 88a137bcfc76410583f910b377d1be7f P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 272b2179c1b4476090a89880df2a8baa (127.9.176.199) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } } }
I20240502 01:54:43.595455 17351 catalog_manager.cc:5484] T 88a137bcfc76410583f910b377d1be7f P 311f765f59754bdcb4eadc04e14bd4ea reported cstate change: config changed from index 6 to 7, NON_VOTER 272b2179c1b4476090a89880df2a8baa (127.9.176.199) added. New cstate: current_term: 1 leader_uuid: "84d3a4b174c34e02aedab1e620731359" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } } }
I20240502 01:54:43.634549 19604 raft_consensus.cc:2932] T 88a137bcfc76410583f910b377d1be7f P 4b7c382c9edd41babdd1a1afc904a813 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 272b2179c1b4476090a89880df2a8baa (127.9.176.199) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } } }
I20240502 01:54:43.691790 19446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:43.702870 22142 raft_consensus.cc:1050] T 2045e08a200148c69326351f38fd484d P 84d3a4b174c34e02aedab1e620731359: attempting to promote NON_VOTER da3584f429104d23a1a8899b35cc2138 to VOTER
I20240502 01:54:43.704766 22142 consensus_queue.cc:237] T 2045e08a200148c69326351f38fd484d P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { promote: false } }
I20240502 01:54:43.725515 17500 consensus_queue.cc:237] T 62fd543d5c7c41f98926f2148ee61e7d P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } }
I20240502 01:54:43.743400 19718 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:43.753638 21844 ts_tablet_manager.cc:1398] T 49ef898a43704b87ba077cd7b0485e9a P 29806d3d37354c169484f6db368f8798: Time spent starting tablet: real 0.184s	user 0.015s	sys 0.016s
I20240502 01:54:43.761830 17520 tablet_copy_service.cc:342] P 311f765f59754bdcb4eadc04e14bd4ea: Request end of tablet copy session 29806d3d37354c169484f6db368f8798-49ef898a43704b87ba077cd7b0485e9a received from {username='slave'} at 127.9.176.197:51011
I20240502 01:54:43.762100 17520 tablet_copy_service.cc:433] P 311f765f59754bdcb4eadc04e14bd4ea: ending tablet copy session 29806d3d37354c169484f6db368f8798-49ef898a43704b87ba077cd7b0485e9a on tablet 49ef898a43704b87ba077cd7b0485e9a with peer 29806d3d37354c169484f6db368f8798
I20240502 01:54:43.791754 19464 raft_consensus.cc:1261] T 2045e08a200148c69326351f38fd484d P 29806d3d37354c169484f6db368f8798 [term 1 FOLLOWER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:43.791796 17746 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:54:43.792984 21362 consensus_queue.cc:1035] T 2045e08a200148c69326351f38fd484d P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:43.801424 17634 raft_consensus.cc:1261] T 2045e08a200148c69326351f38fd484d P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:43.803244 22143 consensus_queue.cc:1035] T 2045e08a200148c69326351f38fd484d P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:43.812850 21467 raft_consensus.cc:1050] T 59e507855e5043d38cef84e92adf437b P 311f765f59754bdcb4eadc04e14bd4ea: attempting to promote NON_VOTER 4b7c382c9edd41babdd1a1afc904a813 to VOTER
I20240502 01:54:43.813989 21467 consensus_queue.cc:237] T 59e507855e5043d38cef84e92adf437b P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } }
I20240502 01:54:43.839864 17764 raft_consensus.cc:1203] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 84d3a4b174c34e02aedab1e620731359 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10]   Dedup: 1.10->[]
I20240502 01:54:43.855486 19604 raft_consensus.cc:1261] T 59e507855e5043d38cef84e92adf437b P 4b7c382c9edd41babdd1a1afc904a813 [term 1 LEARNER]: Refusing update from remote peer 311f765f59754bdcb4eadc04e14bd4ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:43.857432 21938 consensus_queue.cc:1035] T 59e507855e5043d38cef84e92adf437b P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:43.860216 17898 raft_consensus.cc:1261] T 59e507855e5043d38cef84e92adf437b P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Refusing update from remote peer 311f765f59754bdcb4eadc04e14bd4ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:43.861203 17631 raft_consensus.cc:1261] T 59e507855e5043d38cef84e92adf437b P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Refusing update from remote peer 311f765f59754bdcb4eadc04e14bd4ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:43.861311 21938 consensus_queue.cc:1035] T 59e507855e5043d38cef84e92adf437b P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:43.864109 21938 consensus_queue.cc:1035] T 59e507855e5043d38cef84e92adf437b P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:43.884290 21938 raft_consensus.cc:1050] T 37eda67598fe491bbb29aa62b2c4bec7 P 311f765f59754bdcb4eadc04e14bd4ea: attempting to promote NON_VOTER da3584f429104d23a1a8899b35cc2138 to VOTER
I20240502 01:54:43.885704 21938 consensus_queue.cc:237] T 37eda67598fe491bbb29aa62b2c4bec7 P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { promote: false } }
I20240502 01:54:43.900544 17901 raft_consensus.cc:1261] T 2045e08a200148c69326351f38fd484d P da3584f429104d23a1a8899b35cc2138 [term 1 LEARNER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:43.903666 22143 consensus_queue.cc:1035] T 2045e08a200148c69326351f38fd484d P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:43.906831 17478 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:54:43.907847 17433 consensus_peers.cc:485] T 62fd543d5c7c41f98926f2148ee61e7d P 311f765f59754bdcb4eadc04e14bd4ea -> Peer 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200:37831): Couldn't send request to peer 94931f4b51c9493cb3d4264ce99f5e76. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 62fd543d5c7c41f98926f2148ee61e7d. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:43.911808 19736 raft_consensus.cc:1261] T 62fd543d5c7c41f98926f2148ee61e7d P 272b2179c1b4476090a89880df2a8baa [term 1 FOLLOWER]: Refusing update from remote peer 311f765f59754bdcb4eadc04e14bd4ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:43.912618 21409 consensus_queue.cc:1035] T 62fd543d5c7c41f98926f2148ee61e7d P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:43.919785 21844 ts_tablet_manager.cc:902] T a6b5231a18a5414dbd010d4d48f08dbd P 29806d3d37354c169484f6db368f8798: Initiating tablet copy from peer 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193:35397)
I20240502 01:54:43.920033 21409 raft_consensus.cc:2932] T 59e507855e5043d38cef84e92adf437b P 311f765f59754bdcb4eadc04e14bd4ea [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
I20240502 01:54:43.921980 19603 raft_consensus.cc:2932] T 59e507855e5043d38cef84e92adf437b P 4b7c382c9edd41babdd1a1afc904a813 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
I20240502 01:54:43.929554 17352 catalog_manager.cc:5484] T 59e507855e5043d38cef84e92adf437b P 4b7c382c9edd41babdd1a1afc904a813 reported cstate change: config changed from index 4 to 5, 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "311f765f59754bdcb4eadc04e14bd4ea" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
I20240502 01:54:43.941118 19585 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:43.981532 17767 raft_consensus.cc:1261] T 62fd543d5c7c41f98926f2148ee61e7d P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Refusing update from remote peer 311f765f59754bdcb4eadc04e14bd4ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:43.984144 21467 consensus_queue.cc:1035] T 62fd543d5c7c41f98926f2148ee61e7d P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:43.984592 17631 raft_consensus.cc:2932] T 59e507855e5043d38cef84e92adf437b P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
I20240502 01:54:44.018821 19852 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:54:44.019276 17432 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.176.198:36865, user_credentials={real_user=slave}} blocked reactor thread for 92061us
I20240502 01:54:44.020754 22199 raft_consensus.cc:2932] T 62fd543d5c7c41f98926f2148ee61e7d P 311f765f59754bdcb4eadc04e14bd4ea [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } } }
I20240502 01:54:44.021641 17767 raft_consensus.cc:2932] T 62fd543d5c7c41f98926f2148ee61e7d P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } } }
I20240502 01:54:44.030253 17347 catalog_manager.cc:5484] T 62fd543d5c7c41f98926f2148ee61e7d P 84d3a4b174c34e02aedab1e620731359 reported cstate change: config changed from index 6 to 7, NON_VOTER 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) added. New cstate: current_term: 1 leader_uuid: "311f765f59754bdcb4eadc04e14bd4ea" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } } }
I20240502 01:54:44.035490 19869 raft_consensus.cc:1261] T 37eda67598fe491bbb29aa62b2c4bec7 P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Refusing update from remote peer 311f765f59754bdcb4eadc04e14bd4ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 01:54:44.037591 21467 consensus_queue.cc:1035] T 37eda67598fe491bbb29aa62b2c4bec7 P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 01:54:44.062808 22143 raft_consensus.cc:2932] T 2045e08a200148c69326351f38fd484d P 84d3a4b174c34e02aedab1e620731359 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, da3584f429104d23a1a8899b35cc2138 (127.9.176.196) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { promote: false } } }
I20240502 01:54:44.066210 17613 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:54:44.075278 17899 raft_consensus.cc:1261] T 37eda67598fe491bbb29aa62b2c4bec7 P da3584f429104d23a1a8899b35cc2138 [term 1 LEARNER]: Refusing update from remote peer 311f765f59754bdcb4eadc04e14bd4ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 01:54:44.076328 19464 raft_consensus.cc:2932] T 2045e08a200148c69326351f38fd484d P 29806d3d37354c169484f6db368f8798 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, da3584f429104d23a1a8899b35cc2138 (127.9.176.196) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { promote: false } } }
I20240502 01:54:44.076629 17631 raft_consensus.cc:2932] T 2045e08a200148c69326351f38fd484d P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, da3584f429104d23a1a8899b35cc2138 (127.9.176.196) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { promote: false } } }
I20240502 01:54:44.092126 21467 consensus_queue.cc:1035] T 37eda67598fe491bbb29aa62b2c4bec7 P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 01:54:44.094300 19464 raft_consensus.cc:1261] T 37eda67598fe491bbb29aa62b2c4bec7 P 29806d3d37354c169484f6db368f8798 [term 1 FOLLOWER]: Refusing update from remote peer 311f765f59754bdcb4eadc04e14bd4ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 01:54:44.098454 17352 catalog_manager.cc:5484] T 2045e08a200148c69326351f38fd484d P 84d3a4b174c34e02aedab1e620731359 reported cstate change: config changed from index 7 to 8, da3584f429104d23a1a8899b35cc2138 (127.9.176.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "84d3a4b174c34e02aedab1e620731359" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:44.114615 17764 consensus_queue.cc:237] T 2045e08a200148c69326351f38fd484d P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { promote: false } }
I20240502 01:54:44.118369 17901 raft_consensus.cc:1261] T 2045e08a200148c69326351f38fd484d P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:54:44.119086 22143 consensus_queue.cc:1035] T 2045e08a200148c69326351f38fd484d P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:44.120952 17899 raft_consensus.cc:2932] T 2045e08a200148c69326351f38fd484d P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, da3584f429104d23a1a8899b35cc2138 (127.9.176.196) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { promote: false } } }
I20240502 01:54:44.121737 17634 raft_consensus.cc:1261] T 2045e08a200148c69326351f38fd484d P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:54:44.122887 19852 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:44.123114 22143 consensus_queue.cc:1035] T 2045e08a200148c69326351f38fd484d P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:44.132696 22142 raft_consensus.cc:2932] T 2045e08a200148c69326351f38fd484d P 84d3a4b174c34e02aedab1e620731359 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 29806d3d37354c169484f6db368f8798 (127.9.176.197) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { promote: false } } }
I20240502 01:54:44.138990 17343 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2045e08a200148c69326351f38fd484d with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:44.149690 22211 ts_tablet_manager.cc:902] T 88a137bcfc76410583f910b377d1be7f P 272b2179c1b4476090a89880df2a8baa: Initiating tablet copy from peer 84d3a4b174c34e02aedab1e620731359 (127.9.176.195:37357)
I20240502 01:54:44.151463 17631 raft_consensus.cc:2932] T 2045e08a200148c69326351f38fd484d P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 29806d3d37354c169484f6db368f8798 (127.9.176.197) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { promote: false } } }
I20240502 01:54:44.161075 17880 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:54:44.165812 17898 raft_consensus.cc:2932] T 2045e08a200148c69326351f38fd484d P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 29806d3d37354c169484f6db368f8798 (127.9.176.197) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { promote: false } } }
W20240502 01:54:44.171875 17432 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.176.196:46381, user_credentials={real_user=slave}} blocked reactor thread for 81097.9us
I20240502 01:54:44.175748 19464 raft_consensus.cc:1203] T 49ef898a43704b87ba077cd7b0485e9a P 29806d3d37354c169484f6db368f8798 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8]   Dedup: 1.8->[]
I20240502 01:54:44.178565 17500 consensus_queue.cc:237] T 59e507855e5043d38cef84e92adf437b P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } }
I20240502 01:54:44.182703 17631 raft_consensus.cc:1261] T 59e507855e5043d38cef84e92adf437b P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Refusing update from remote peer 311f765f59754bdcb4eadc04e14bd4ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:44.183629 21938 consensus_queue.cc:1035] T 59e507855e5043d38cef84e92adf437b P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:44.185420 17347 catalog_manager.cc:5484] T 2045e08a200148c69326351f38fd484d P 84d3a4b174c34e02aedab1e620731359 reported cstate change: config changed from index 8 to 9, VOTER 29806d3d37354c169484f6db368f8798 (127.9.176.197) evicted. New cstate: current_term: 1 leader_uuid: "84d3a4b174c34e02aedab1e620731359" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:44.189721 21467 raft_consensus.cc:2932] T 59e507855e5043d38cef84e92adf437b P 311f765f59754bdcb4eadc04e14bd4ea [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER da3584f429104d23a1a8899b35cc2138 (127.9.176.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
I20240502 01:54:44.195962 17631 raft_consensus.cc:2932] T 59e507855e5043d38cef84e92adf437b P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER da3584f429104d23a1a8899b35cc2138 (127.9.176.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
I20240502 01:54:44.198544 19446 tablet_service.cc:1514] Processing DeleteTablet for tablet 2045e08a200148c69326351f38fd484d with delete_type TABLET_DATA_TOMBSTONED (TS 29806d3d37354c169484f6db368f8798 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:33166
I20240502 01:54:44.208352 22227 tablet_replica.cc:330] T 2045e08a200148c69326351f38fd484d P 29806d3d37354c169484f6db368f8798: stopping tablet replica
I20240502 01:54:44.208832 22227 raft_consensus.cc:2229] T 2045e08a200148c69326351f38fd484d P 29806d3d37354c169484f6db368f8798 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:44.209383 22227 raft_consensus.cc:2258] T 2045e08a200148c69326351f38fd484d P 29806d3d37354c169484f6db368f8798 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:44.210665 22227 ts_tablet_manager.cc:1875] T 2045e08a200148c69326351f38fd484d P 29806d3d37354c169484f6db368f8798: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:44.232432 22211 tablet_copy_client.cc:308] T 88a137bcfc76410583f910b377d1be7f P 272b2179c1b4476090a89880df2a8baa: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.195:37357
I20240502 01:54:44.246297 17899 raft_consensus.cc:2932] T 59e507855e5043d38cef84e92adf437b P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
I20240502 01:54:44.255439 17789 tablet_copy_service.cc:140] P 84d3a4b174c34e02aedab1e620731359: Received BeginTabletCopySession request for tablet 88a137bcfc76410583f910b377d1be7f from peer 272b2179c1b4476090a89880df2a8baa ({username='slave'} at 127.9.176.199:46159)
I20240502 01:54:44.255678 17789 tablet_copy_service.cc:161] P 84d3a4b174c34e02aedab1e620731359: Beginning new tablet copy session on tablet 88a137bcfc76410583f910b377d1be7f from peer 272b2179c1b4476090a89880df2a8baa at {username='slave'} at 127.9.176.199:46159: session id = 272b2179c1b4476090a89880df2a8baa-88a137bcfc76410583f910b377d1be7f
I20240502 01:54:44.258878 17789 tablet_copy_source_session.cc:205] T 88a137bcfc76410583f910b377d1be7f P 84d3a4b174c34e02aedab1e620731359: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:44.262493 17478 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:54:44.262797 17613 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:54:44.263247 22211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88a137bcfc76410583f910b377d1be7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:44.280360 22211 tablet_copy_client.cc:783] T 88a137bcfc76410583f910b377d1be7f P 272b2179c1b4476090a89880df2a8baa: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:44.280709 22211 tablet_copy_client.cc:647] T 88a137bcfc76410583f910b377d1be7f P 272b2179c1b4476090a89880df2a8baa: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:44.292032 22211 tablet_copy_client.cc:523] T 88a137bcfc76410583f910b377d1be7f P 272b2179c1b4476090a89880df2a8baa: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:44.296226 19736 raft_consensus.cc:2932] T 62fd543d5c7c41f98926f2148ee61e7d P 272b2179c1b4476090a89880df2a8baa [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } } }
I20240502 01:54:44.298420 19718 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:44.300343 22211 tablet_bootstrap.cc:492] T 88a137bcfc76410583f910b377d1be7f P 272b2179c1b4476090a89880df2a8baa: Bootstrap starting.
I20240502 01:54:44.309216 17347 catalog_manager.cc:5484] T 59e507855e5043d38cef84e92adf437b P 1b50a031f2784ba498b3e52e16916205 reported cstate change: config changed from index 5 to 6, VOTER da3584f429104d23a1a8899b35cc2138 (127.9.176.196) evicted. New cstate: current_term: 1 leader_uuid: "311f765f59754bdcb4eadc04e14bd4ea" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
I20240502 01:54:44.314838 19446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:44.316417 17478 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:54:44.327864 17880 tablet_service.cc:1514] Processing DeleteTablet for tablet 59e507855e5043d38cef84e92adf437b with delete_type TABLET_DATA_TOMBSTONED (TS da3584f429104d23a1a8899b35cc2138 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:34396
I20240502 01:54:44.342553 19852 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:54:44.346057 17432 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.176.197:35011, user_credentials={real_user=slave}} blocked reactor thread for 50830.7us
I20240502 01:54:44.362056 22218 consensus_queue.cc:1035] T 37eda67598fe491bbb29aa62b2c4bec7 P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 01:54:44.363868 21844 tablet_copy_client.cc:308] T a6b5231a18a5414dbd010d4d48f08dbd P 29806d3d37354c169484f6db368f8798: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.193:35397
I20240502 01:54:44.364738 17520 tablet_copy_service.cc:140] P 311f765f59754bdcb4eadc04e14bd4ea: Received BeginTabletCopySession request for tablet a6b5231a18a5414dbd010d4d48f08dbd from peer 29806d3d37354c169484f6db368f8798 ({username='slave'} at 127.9.176.197:51011)
I20240502 01:54:44.364985 17520 tablet_copy_service.cc:161] P 311f765f59754bdcb4eadc04e14bd4ea: Beginning new tablet copy session on tablet a6b5231a18a5414dbd010d4d48f08dbd from peer 29806d3d37354c169484f6db368f8798 at {username='slave'} at 127.9.176.197:51011: session id = 29806d3d37354c169484f6db368f8798-a6b5231a18a5414dbd010d4d48f08dbd
I20240502 01:54:44.369397 17520 tablet_copy_source_session.cc:205] T a6b5231a18a5414dbd010d4d48f08dbd P 311f765f59754bdcb4eadc04e14bd4ea: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:44.369555 22052 raft_consensus.cc:1050] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P da3584f429104d23a1a8899b35cc2138: attempting to promote NON_VOTER 84d3a4b174c34e02aedab1e620731359 to VOTER
I20240502 01:54:44.370605 22052 consensus_queue.cc:237] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P da3584f429104d23a1a8899b35cc2138 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { promote: false } }
I20240502 01:54:44.393317 21844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6b5231a18a5414dbd010d4d48f08dbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:44.401840 19604 raft_consensus.cc:1261] T 59e507855e5043d38cef84e92adf437b P 4b7c382c9edd41babdd1a1afc904a813 [term 1 FOLLOWER]: Refusing update from remote peer 311f765f59754bdcb4eadc04e14bd4ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:44.402911 22246 tablet_replica.cc:330] T 59e507855e5043d38cef84e92adf437b P da3584f429104d23a1a8899b35cc2138: stopping tablet replica
I20240502 01:54:44.403162 22246 raft_consensus.cc:2229] T 59e507855e5043d38cef84e92adf437b P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:44.403435 22246 raft_consensus.cc:2258] T 59e507855e5043d38cef84e92adf437b P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:44.405349 17343 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 59e507855e5043d38cef84e92adf437b with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:44.406828 19585 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:44.408160 17746 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:54:44.393355 22227 ts_tablet_manager.cc:1888] T 2045e08a200148c69326351f38fd484d P 29806d3d37354c169484f6db368f8798: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 01:54:44.415967 19585 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:44.429364 22246 ts_tablet_manager.cc:1875] T 59e507855e5043d38cef84e92adf437b P da3584f429104d23a1a8899b35cc2138: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:44.430577 22197 consensus_queue.cc:1035] T 59e507855e5043d38cef84e92adf437b P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:44.435636 21938 raft_consensus.cc:2932] T 37eda67598fe491bbb29aa62b2c4bec7 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, da3584f429104d23a1a8899b35cc2138 (127.9.176.196) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { promote: false } } }
I20240502 01:54:44.442392 19603 raft_consensus.cc:2932] T 59e507855e5043d38cef84e92adf437b P 4b7c382c9edd41babdd1a1afc904a813 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER da3584f429104d23a1a8899b35cc2138 (127.9.176.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
I20240502 01:54:44.448161 22227 log.cc:1196] T 2045e08a200148c69326351f38fd484d P 29806d3d37354c169484f6db368f8798: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/wal/wals/2045e08a200148c69326351f38fd484d
I20240502 01:54:44.460279 22248 ts_tablet_manager.cc:902] T 62fd543d5c7c41f98926f2148ee61e7d P 94931f4b51c9493cb3d4264ce99f5e76: Initiating tablet copy from peer 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193:35397)
I20240502 01:54:44.463269 17767 raft_consensus.cc:1261] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 84d3a4b174c34e02aedab1e620731359 [term 1 LEARNER]: Refusing update from remote peer da3584f429104d23a1a8899b35cc2138: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 01:54:44.464092 19736 raft_consensus.cc:1261] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 272b2179c1b4476090a89880df2a8baa [term 1 FOLLOWER]: Refusing update from remote peer da3584f429104d23a1a8899b35cc2138: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 01:54:44.464290 22052 consensus_queue.cc:1035] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P da3584f429104d23a1a8899b35cc2138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 01:54:44.460861 19869 raft_consensus.cc:1261] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Refusing update from remote peer da3584f429104d23a1a8899b35cc2138: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 01:54:44.468726 22052 consensus_queue.cc:1035] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P da3584f429104d23a1a8899b35cc2138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 01:54:44.488307 22246 ts_tablet_manager.cc:1888] T 59e507855e5043d38cef84e92adf437b P da3584f429104d23a1a8899b35cc2138: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:54:44.472245 19718 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:44.502568 22211 tablet_bootstrap.cc:492] T 88a137bcfc76410583f910b377d1be7f P 272b2179c1b4476090a89880df2a8baa: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:44.503247 22211 tablet_bootstrap.cc:492] T 88a137bcfc76410583f910b377d1be7f P 272b2179c1b4476090a89880df2a8baa: Bootstrap complete.
I20240502 01:54:44.504793 17347 catalog_manager.cc:5484] T 37eda67598fe491bbb29aa62b2c4bec7 P 311f765f59754bdcb4eadc04e14bd4ea reported cstate change: config changed from index 12 to 13, da3584f429104d23a1a8899b35cc2138 (127.9.176.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "311f765f59754bdcb4eadc04e14bd4ea" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:44.503751 22211 ts_tablet_manager.cc:1367] T 88a137bcfc76410583f910b377d1be7f P 272b2179c1b4476090a89880df2a8baa: Time spent bootstrapping tablet: real 0.203s	user 0.065s	sys 0.000s
I20240502 01:54:44.508865 22211 raft_consensus.cc:348] T 88a137bcfc76410583f910b377d1be7f P 272b2179c1b4476090a89880df2a8baa [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } }
I20240502 01:54:44.509366 22211 raft_consensus.cc:729] T 88a137bcfc76410583f910b377d1be7f P 272b2179c1b4476090a89880df2a8baa [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 272b2179c1b4476090a89880df2a8baa, State: Initialized, Role: LEARNER
I20240502 01:54:44.509657 22211 consensus_queue.cc:260] T 88a137bcfc76410583f910b377d1be7f P 272b2179c1b4476090a89880df2a8baa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } }
I20240502 01:54:44.510650 22211 ts_tablet_manager.cc:1398] T 88a137bcfc76410583f910b377d1be7f P 272b2179c1b4476090a89880df2a8baa: Time spent starting tablet: real 0.003s	user 0.003s	sys 0.000s
I20240502 01:54:44.512713 19869 raft_consensus.cc:2932] T 37eda67598fe491bbb29aa62b2c4bec7 P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, da3584f429104d23a1a8899b35cc2138 (127.9.176.196) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { promote: false } } }
I20240502 01:54:44.518442 17789 tablet_copy_service.cc:342] P 84d3a4b174c34e02aedab1e620731359: Request end of tablet copy session 272b2179c1b4476090a89880df2a8baa-88a137bcfc76410583f910b377d1be7f received from {username='slave'} at 127.9.176.199:46159
I20240502 01:54:44.518635 17789 tablet_copy_service.cc:433] P 84d3a4b174c34e02aedab1e620731359: ending tablet copy session 272b2179c1b4476090a89880df2a8baa-88a137bcfc76410583f910b377d1be7f on tablet 88a137bcfc76410583f910b377d1be7f with peer 272b2179c1b4476090a89880df2a8baa
I20240502 01:54:44.531775 17746 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:54:44.534698 21844 tablet_copy_client.cc:783] T a6b5231a18a5414dbd010d4d48f08dbd P 29806d3d37354c169484f6db368f8798: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:44.535372 21844 tablet_copy_client.cc:647] T a6b5231a18a5414dbd010d4d48f08dbd P 29806d3d37354c169484f6db368f8798: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:44.538981 17499 consensus_queue.cc:237] T 37eda67598fe491bbb29aa62b2c4bec7 P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 1.13, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { promote: false } }
I20240502 01:54:44.542821 21844 tablet_copy_client.cc:523] T a6b5231a18a5414dbd010d4d48f08dbd P 29806d3d37354c169484f6db368f8798: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:44.545362 17500 consensus_queue.cc:237] T fadcc8fe6ae047c1b142514aad8bebba P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } }
I20240502 01:54:44.557339 19870 raft_consensus.cc:1261] T 37eda67598fe491bbb29aa62b2c4bec7 P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Refusing update from remote peer 311f765f59754bdcb4eadc04e14bd4ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240502 01:54:44.560092 22218 consensus_queue.cc:1035] T 37eda67598fe491bbb29aa62b2c4bec7 P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240502 01:54:44.571818 22218 raft_consensus.cc:1050] T 49ef898a43704b87ba077cd7b0485e9a P 311f765f59754bdcb4eadc04e14bd4ea: attempting to promote NON_VOTER 29806d3d37354c169484f6db368f8798 to VOTER
I20240502 01:54:44.574497 17767 raft_consensus.cc:1261] T fadcc8fe6ae047c1b142514aad8bebba P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Refusing update from remote peer 311f765f59754bdcb4eadc04e14bd4ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:44.575495 21467 consensus_queue.cc:1035] T fadcc8fe6ae047c1b142514aad8bebba P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:44.577426 22218 consensus_queue.cc:237] T 49ef898a43704b87ba077cd7b0485e9a P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } }
W20240502 01:54:44.581795 17432 consensus_peers.cc:485] T fadcc8fe6ae047c1b142514aad8bebba P 311f765f59754bdcb4eadc04e14bd4ea -> Peer 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198:36865): Couldn't send request to peer 4b7c382c9edd41babdd1a1afc904a813. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fadcc8fe6ae047c1b142514aad8bebba. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:44.584509 17613 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:54:44.592702 21467 raft_consensus.cc:2932] T 37eda67598fe491bbb29aa62b2c4bec7 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 LEADER]: Committing config change with OpId 1.14: config changed from index 13 to 14, VOTER 29806d3d37354c169484f6db368f8798 (127.9.176.197) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { promote: false } } }
I20240502 01:54:44.490044 22052 consensus_queue.cc:1035] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P da3584f429104d23a1a8899b35cc2138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 01:54:44.597550 21979 raft_consensus.cc:2932] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P da3584f429104d23a1a8899b35cc2138 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 84d3a4b174c34e02aedab1e620731359 (127.9.176.195) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { promote: false } } }
I20240502 01:54:44.599076 19870 raft_consensus.cc:2932] T 37eda67598fe491bbb29aa62b2c4bec7 P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, VOTER 29806d3d37354c169484f6db368f8798 (127.9.176.197) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { promote: false } } }
I20240502 01:54:44.600466 19872 raft_consensus.cc:2932] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 84d3a4b174c34e02aedab1e620731359 (127.9.176.195) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { promote: false } } }
I20240502 01:54:44.494844 17901 raft_consensus.cc:2932] T 37eda67598fe491bbb29aa62b2c4bec7 P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, da3584f429104d23a1a8899b35cc2138 (127.9.176.196) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { promote: false } } }
I20240502 01:54:44.606755 17340 catalog_manager.cc:4830] TS 29806d3d37354c169484f6db368f8798 (127.9.176.197:35011): tablet 2045e08a200148c69326351f38fd484d (table rebalance_test_table_1 [id=952912d7ec8d493bbb8985fffe9aa918]) successfully deleted
I20240502 01:54:44.607136 17343 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 37eda67598fe491bbb29aa62b2c4bec7 with cas_config_opid_index 13: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:44.621717 22246 log.cc:1196] T 59e507855e5043d38cef84e92adf437b P da3584f429104d23a1a8899b35cc2138: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/wal/wals/59e507855e5043d38cef84e92adf437b
I20240502 01:54:44.622758 21844 tablet_bootstrap.cc:492] T a6b5231a18a5414dbd010d4d48f08dbd P 29806d3d37354c169484f6db368f8798: Bootstrap starting.
W20240502 01:54:44.630821 17835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.176.195:37357, user_credentials={real_user=slave}} blocked reactor thread for 131311us
I20240502 01:54:44.632313 17340 catalog_manager.cc:4830] TS da3584f429104d23a1a8899b35cc2138 (127.9.176.196:46381): tablet 59e507855e5043d38cef84e92adf437b (table rebalance_test_table_4 [id=14b25b3f4ede4b65870815b014d90449]) successfully deleted
I20240502 01:54:44.635033 17901 raft_consensus.cc:1261] T 37eda67598fe491bbb29aa62b2c4bec7 P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Refusing update from remote peer 311f765f59754bdcb4eadc04e14bd4ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240502 01:54:44.635859 17901 raft_consensus.cc:1261] T fadcc8fe6ae047c1b142514aad8bebba P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Refusing update from remote peer 311f765f59754bdcb4eadc04e14bd4ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:44.638711 22197 consensus_queue.cc:1035] T 37eda67598fe491bbb29aa62b2c4bec7 P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.002s
I20240502 01:54:44.642050 22197 consensus_queue.cc:1035] T fadcc8fe6ae047c1b142514aad8bebba P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:44.647557 17880 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:54:44.647351 17901 raft_consensus.cc:2932] T 37eda67598fe491bbb29aa62b2c4bec7 P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, VOTER 29806d3d37354c169484f6db368f8798 (127.9.176.197) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { promote: false } } }
I20240502 01:54:44.657943 19446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:44.658007 21467 raft_consensus.cc:2932] T fadcc8fe6ae047c1b142514aad8bebba P 311f765f59754bdcb4eadc04e14bd4ea [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } } }
I20240502 01:54:44.661299 17880 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:54:44.661602 19446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:44.662647 19718 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:44.667382 17900 raft_consensus.cc:2932] T fadcc8fe6ae047c1b142514aad8bebba P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } } }
I20240502 01:54:44.668404 17767 raft_consensus.cc:2932] T fadcc8fe6ae047c1b142514aad8bebba P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } } }
I20240502 01:54:44.675489 17348 catalog_manager.cc:5484] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P da3584f429104d23a1a8899b35cc2138 reported cstate change: config changed from index 10 to 11, 84d3a4b174c34e02aedab1e620731359 (127.9.176.195) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "da3584f429104d23a1a8899b35cc2138" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:44.686210 17347 catalog_manager.cc:5484] T 37eda67598fe491bbb29aa62b2c4bec7 P 94931f4b51c9493cb3d4264ce99f5e76 reported cstate change: config changed from index 13 to 14, VOTER 29806d3d37354c169484f6db368f8798 (127.9.176.197) evicted. New cstate: current_term: 1 leader_uuid: "311f765f59754bdcb4eadc04e14bd4ea" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { promote: false } } }
I20240502 01:54:44.692435 19446 tablet_service.cc:1514] Processing DeleteTablet for tablet 37eda67598fe491bbb29aa62b2c4bec7 with delete_type TABLET_DATA_TOMBSTONED (TS 29806d3d37354c169484f6db368f8798 not found in new config with opid_index 14) from {username='slave'} at 127.0.0.1:33166
I20240502 01:54:44.693759 17352 catalog_manager.cc:5484] T fadcc8fe6ae047c1b142514aad8bebba P 84d3a4b174c34e02aedab1e620731359 reported cstate change: config changed from index -1 to 4, NON_VOTER 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) added. New cstate: current_term: 1 leader_uuid: "311f765f59754bdcb4eadc04e14bd4ea" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } } }
I20240502 01:54:44.695978 19736 raft_consensus.cc:2932] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 272b2179c1b4476090a89880df2a8baa [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 84d3a4b174c34e02aedab1e620731359 (127.9.176.195) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { promote: false } } }
I20240502 01:54:44.702322 22227 tablet_replica.cc:330] T 37eda67598fe491bbb29aa62b2c4bec7 P 29806d3d37354c169484f6db368f8798: stopping tablet replica
I20240502 01:54:44.703058 22227 raft_consensus.cc:2229] T 37eda67598fe491bbb29aa62b2c4bec7 P 29806d3d37354c169484f6db368f8798 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:44.703763 22227 pending_rounds.cc:62] T 37eda67598fe491bbb29aa62b2c4bec7 P 29806d3d37354c169484f6db368f8798: Trying to abort 1 pending ops.
I20240502 01:54:44.704891 22227 pending_rounds.cc:68] T 37eda67598fe491bbb29aa62b2c4bec7 P 29806d3d37354c169484f6db368f8798: Aborting op as it isn't in flight: id { term: 1 index: 13 } timestamp: 7023062564392431616 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "37eda67598fe491bbb29aa62b2c4bec7" old_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { promote: true } } } new_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { promote: false } } } }
I20240502 01:54:44.708076 22227 raft_consensus.cc:2894] T 37eda67598fe491bbb29aa62b2c4bec7 P 29806d3d37354c169484f6db368f8798 [term 1 FOLLOWER]: Aborting config change with OpId 1.13: Aborted: Op aborted
I20240502 01:54:44.708873 22227 raft_consensus.cc:2866] T 37eda67598fe491bbb29aa62b2c4bec7 P 29806d3d37354c169484f6db368f8798 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240502 01:54:44.711490 22227 raft_consensus.cc:2258] T 37eda67598fe491bbb29aa62b2c4bec7 P 29806d3d37354c169484f6db368f8798 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240502 01:54:44.723670 17833 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.176.200:37831, user_credentials={real_user=slave}} blocked reactor thread for 44413.3us
I20240502 01:54:44.732194 19464 raft_consensus.cc:1261] T 49ef898a43704b87ba077cd7b0485e9a P 29806d3d37354c169484f6db368f8798 [term 1 LEARNER]: Refusing update from remote peer 311f765f59754bdcb4eadc04e14bd4ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:54:44.733590 19872 raft_consensus.cc:1261] T 49ef898a43704b87ba077cd7b0485e9a P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Refusing update from remote peer 311f765f59754bdcb4eadc04e14bd4ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:54:44.733666 17897 raft_consensus.cc:1261] T 49ef898a43704b87ba077cd7b0485e9a P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Refusing update from remote peer 311f765f59754bdcb4eadc04e14bd4ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:54:44.734406 21467 consensus_queue.cc:1035] T 49ef898a43704b87ba077cd7b0485e9a P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:44.735129 21467 consensus_queue.cc:1035] T 49ef898a43704b87ba077cd7b0485e9a P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:44.739861 22218 consensus_queue.cc:1035] T 49ef898a43704b87ba077cd7b0485e9a P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:44.756440 22227 ts_tablet_manager.cc:1875] T 37eda67598fe491bbb29aa62b2c4bec7 P 29806d3d37354c169484f6db368f8798: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:44.761318 21467 raft_consensus.cc:2932] T 49ef898a43704b87ba077cd7b0485e9a P 311f765f59754bdcb4eadc04e14bd4ea [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 29806d3d37354c169484f6db368f8798 (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } }
I20240502 01:54:44.771700 22248 tablet_copy_client.cc:308] T 62fd543d5c7c41f98926f2148ee61e7d P 94931f4b51c9493cb3d4264ce99f5e76: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.193:35397
I20240502 01:54:44.771911 19467 raft_consensus.cc:2932] T 49ef898a43704b87ba077cd7b0485e9a P 29806d3d37354c169484f6db368f8798 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 29806d3d37354c169484f6db368f8798 (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } }
I20240502 01:54:44.772706 17520 tablet_copy_service.cc:140] P 311f765f59754bdcb4eadc04e14bd4ea: Received BeginTabletCopySession request for tablet 62fd543d5c7c41f98926f2148ee61e7d from peer 94931f4b51c9493cb3d4264ce99f5e76 ({username='slave'} at 127.9.176.200:46111)
I20240502 01:54:44.772877 17520 tablet_copy_service.cc:161] P 311f765f59754bdcb4eadc04e14bd4ea: Beginning new tablet copy session on tablet 62fd543d5c7c41f98926f2148ee61e7d from peer 94931f4b51c9493cb3d4264ce99f5e76 at {username='slave'} at 127.9.176.200:46111: session id = 94931f4b51c9493cb3d4264ce99f5e76-62fd543d5c7c41f98926f2148ee61e7d
I20240502 01:54:44.776454 17520 tablet_copy_source_session.cc:205] T 62fd543d5c7c41f98926f2148ee61e7d P 311f765f59754bdcb4eadc04e14bd4ea: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:44.781523 17478 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:54:44.788338 22248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62fd543d5c7c41f98926f2148ee61e7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:44.791617 19872 raft_consensus.cc:2932] T 49ef898a43704b87ba077cd7b0485e9a P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 29806d3d37354c169484f6db368f8798 (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } }
I20240502 01:54:44.799605 17351 catalog_manager.cc:5484] T 49ef898a43704b87ba077cd7b0485e9a P 311f765f59754bdcb4eadc04e14bd4ea reported cstate change: config changed from index 8 to 9, 29806d3d37354c169484f6db368f8798 (127.9.176.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "311f765f59754bdcb4eadc04e14bd4ea" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:44.804214 17897 raft_consensus.cc:2932] T 49ef898a43704b87ba077cd7b0485e9a P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 29806d3d37354c169484f6db368f8798 (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } }
I20240502 01:54:44.804934 17674 rpcz_store.cc:275] Call kudu.server.GenericService.GetFlags from 127.0.0.1:55470 (request call id 6) took 1087 ms. Trace:
I20240502 01:54:44.805212 17674 rpcz_store.cc:276] 0502 01:54:43.717624 (+     0us) service_pool.cc:168] Inserting onto call queue
0502 01:54:43.717746 (+   122us) service_pool.cc:225] Handling call
0502 01:54:44.804755 (+1087009us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 01:54:44.824612 19585 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:44.862601 22248 tablet_copy_client.cc:783] T 62fd543d5c7c41f98926f2148ee61e7d P 94931f4b51c9493cb3d4264ce99f5e76: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:44.862592 19852 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:44.862991 22248 tablet_copy_client.cc:647] T 62fd543d5c7c41f98926f2148ee61e7d P 94931f4b51c9493cb3d4264ce99f5e76: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:44.879828 22248 tablet_copy_client.cc:523] T 62fd543d5c7c41f98926f2148ee61e7d P 94931f4b51c9493cb3d4264ce99f5e76: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:44.884888 17767 raft_consensus.cc:2932] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 84d3a4b174c34e02aedab1e620731359 (127.9.176.195) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { promote: false } } }
I20240502 01:54:44.897058 17631 consensus_queue.cc:237] T 38a60d7db47149ff90444e5276fc8935 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } }
I20240502 01:54:44.901638 17500 raft_consensus.cc:1261] T 38a60d7db47149ff90444e5276fc8935 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:54:44.903659 22086 consensus_queue.cc:1035] T 38a60d7db47149ff90444e5276fc8935 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:44.904251 17897 consensus_queue.cc:237] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P da3584f429104d23a1a8899b35cc2138 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { promote: false } }
I20240502 01:54:44.906828 19736 raft_consensus.cc:1261] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 272b2179c1b4476090a89880df2a8baa [term 1 FOLLOWER]: Refusing update from remote peer da3584f429104d23a1a8899b35cc2138: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 01:54:44.907511 22052 consensus_queue.cc:1035] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P da3584f429104d23a1a8899b35cc2138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 01:54:44.908481 19869 raft_consensus.cc:1261] T 38a60d7db47149ff90444e5276fc8935 P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:54:44.911192 17764 raft_consensus.cc:1261] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Refusing update from remote peer da3584f429104d23a1a8899b35cc2138: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 01:54:44.911933 22052 consensus_queue.cc:1035] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P da3584f429104d23a1a8899b35cc2138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 01:54:44.914800 22276 raft_consensus.cc:2932] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P da3584f429104d23a1a8899b35cc2138 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { promote: false } } }
I20240502 01:54:44.917235 17764 raft_consensus.cc:2932] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { promote: false } } }
I20240502 01:54:44.920111 19736 raft_consensus.cc:2932] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 272b2179c1b4476090a89880df2a8baa [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { promote: false } } }
W20240502 01:54:44.924203 17565 consensus_peers.cc:485] T 38a60d7db47149ff90444e5276fc8935 P 1b50a031f2784ba498b3e52e16916205 -> Peer 29806d3d37354c169484f6db368f8798 (127.9.176.197:35011): Couldn't send request to peer 29806d3d37354c169484f6db368f8798. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 38a60d7db47149ff90444e5276fc8935. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:44.926712 17340 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 60ff2e50b1d74e0c8e0890b87f9ac8ba with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:44.928710 17347 catalog_manager.cc:5484] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P da3584f429104d23a1a8899b35cc2138 reported cstate change: config changed from index 11 to 12, VOTER 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) evicted. New cstate: current_term: 1 leader_uuid: "da3584f429104d23a1a8899b35cc2138" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:44.935534 22086 consensus_queue.cc:1035] T 38a60d7db47149ff90444e5276fc8935 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:44.944427 22227 ts_tablet_manager.cc:1888] T 37eda67598fe491bbb29aa62b2c4bec7 P 29806d3d37354c169484f6db368f8798: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.13
I20240502 01:54:44.944676 22227 log.cc:1196] T 37eda67598fe491bbb29aa62b2c4bec7 P 29806d3d37354c169484f6db368f8798: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/wal/wals/37eda67598fe491bbb29aa62b2c4bec7
I20240502 01:54:44.945593 17340 catalog_manager.cc:4830] TS 29806d3d37354c169484f6db368f8798 (127.9.176.197:35011): tablet 37eda67598fe491bbb29aa62b2c4bec7 (table rebalance_test_table_1 [id=952912d7ec8d493bbb8985fffe9aa918]) successfully deleted
I20240502 01:54:44.956620 17746 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:54:44.982156 22086 raft_consensus.cc:2932] T 38a60d7db47149ff90444e5276fc8935 P 1b50a031f2784ba498b3e52e16916205 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 29806d3d37354c169484f6db368f8798 (127.9.176.197) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } } }
I20240502 01:54:44.984457 22248 tablet_bootstrap.cc:492] T 62fd543d5c7c41f98926f2148ee61e7d P 94931f4b51c9493cb3d4264ce99f5e76: Bootstrap starting.
I20240502 01:54:44.989862 17500 raft_consensus.cc:2932] T 38a60d7db47149ff90444e5276fc8935 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 29806d3d37354c169484f6db368f8798 (127.9.176.197) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } } }
I20240502 01:54:44.990048 17347 catalog_manager.cc:5484] T 38a60d7db47149ff90444e5276fc8935 P 1b50a031f2784ba498b3e52e16916205 reported cstate change: config changed from index 8 to 9, NON_VOTER 29806d3d37354c169484f6db368f8798 (127.9.176.197) added. New cstate: current_term: 1 leader_uuid: "1b50a031f2784ba498b3e52e16916205" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:45.006614 19870 raft_consensus.cc:2932] T 38a60d7db47149ff90444e5276fc8935 P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 29806d3d37354c169484f6db368f8798 (127.9.176.197) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } } }
I20240502 01:54:45.045132 19852 tablet_service.cc:1514] Processing DeleteTablet for tablet 60ff2e50b1d74e0c8e0890b87f9ac8ba with delete_type TABLET_DATA_TOMBSTONED (TS 94931f4b51c9493cb3d4264ce99f5e76 not found in new config with opid_index 12) from {username='slave'} at 127.0.0.1:49690
I20240502 01:54:45.066782 17880 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:54:45.075542 19736 raft_consensus.cc:1203] T 88a137bcfc76410583f910b377d1be7f P 272b2179c1b4476090a89880df2a8baa [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240502 01:54:45.077008 22302 tablet_replica.cc:330] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 94931f4b51c9493cb3d4264ce99f5e76: stopping tablet replica
I20240502 01:54:45.079229 22302 raft_consensus.cc:2229] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:45.080471 22302 raft_consensus.cc:2258] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:45.083245 22302 ts_tablet_manager.cc:1875] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 94931f4b51c9493cb3d4264ce99f5e76: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:45.101773 22302 ts_tablet_manager.cc:1888] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 94931f4b51c9493cb3d4264ce99f5e76: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.11
I20240502 01:54:45.102654 22302 log.cc:1196] T 60ff2e50b1d74e0c8e0890b87f9ac8ba P 94931f4b51c9493cb3d4264ce99f5e76: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-7/wal/wals/60ff2e50b1d74e0c8e0890b87f9ac8ba
I20240502 01:54:45.105159 17341 catalog_manager.cc:4830] TS 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200:37831): tablet 60ff2e50b1d74e0c8e0890b87f9ac8ba (table rebalance_test_table_6 [id=d49bf19a92cf45ceadf02d01dbdae6ad]) successfully deleted
I20240502 01:54:45.164921 22311 ts_tablet_manager.cc:902] T fadcc8fe6ae047c1b142514aad8bebba P 4b7c382c9edd41babdd1a1afc904a813: Initiating tablet copy from peer 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193:35397)
I20240502 01:54:45.219503 22248 tablet_bootstrap.cc:492] T 62fd543d5c7c41f98926f2148ee61e7d P 94931f4b51c9493cb3d4264ce99f5e76: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:45.230813 22248 tablet_bootstrap.cc:492] T 62fd543d5c7c41f98926f2148ee61e7d P 94931f4b51c9493cb3d4264ce99f5e76: Bootstrap complete.
I20240502 01:54:45.231243 22248 ts_tablet_manager.cc:1367] T 62fd543d5c7c41f98926f2148ee61e7d P 94931f4b51c9493cb3d4264ce99f5e76: Time spent bootstrapping tablet: real 0.247s	user 0.056s	sys 0.005s
I20240502 01:54:45.232268 22248 raft_consensus.cc:348] T 62fd543d5c7c41f98926f2148ee61e7d P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } }
I20240502 01:54:45.232589 22248 raft_consensus.cc:729] T 62fd543d5c7c41f98926f2148ee61e7d P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 94931f4b51c9493cb3d4264ce99f5e76, State: Initialized, Role: LEARNER
I20240502 01:54:45.232815 22248 consensus_queue.cc:260] T 62fd543d5c7c41f98926f2148ee61e7d P 94931f4b51c9493cb3d4264ce99f5e76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: NON_VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: true } }
I20240502 01:54:45.233457 22248 ts_tablet_manager.cc:1398] T 62fd543d5c7c41f98926f2148ee61e7d P 94931f4b51c9493cb3d4264ce99f5e76: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240502 01:54:45.240335 17520 tablet_copy_service.cc:342] P 311f765f59754bdcb4eadc04e14bd4ea: Request end of tablet copy session 94931f4b51c9493cb3d4264ce99f5e76-62fd543d5c7c41f98926f2148ee61e7d received from {username='slave'} at 127.9.176.200:46111
I20240502 01:54:45.240504 17520 tablet_copy_service.cc:433] P 311f765f59754bdcb4eadc04e14bd4ea: ending tablet copy session 94931f4b51c9493cb3d4264ce99f5e76-62fd543d5c7c41f98926f2148ee61e7d on tablet 62fd543d5c7c41f98926f2148ee61e7d with peer 94931f4b51c9493cb3d4264ce99f5e76
I20240502 01:54:45.256801 21844 tablet_bootstrap.cc:492] T a6b5231a18a5414dbd010d4d48f08dbd P 29806d3d37354c169484f6db368f8798: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:45.257387 21844 tablet_bootstrap.cc:492] T a6b5231a18a5414dbd010d4d48f08dbd P 29806d3d37354c169484f6db368f8798: Bootstrap complete.
I20240502 01:54:45.257737 21844 ts_tablet_manager.cc:1367] T a6b5231a18a5414dbd010d4d48f08dbd P 29806d3d37354c169484f6db368f8798: Time spent bootstrapping tablet: real 0.635s	user 0.050s	sys 0.015s
I20240502 01:54:45.258644 21844 raft_consensus.cc:348] T a6b5231a18a5414dbd010d4d48f08dbd P 29806d3d37354c169484f6db368f8798 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } }
I20240502 01:54:45.258973 21844 raft_consensus.cc:729] T a6b5231a18a5414dbd010d4d48f08dbd P 29806d3d37354c169484f6db368f8798 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 29806d3d37354c169484f6db368f8798, State: Initialized, Role: LEARNER
I20240502 01:54:45.259217 21844 consensus_queue.cc:260] T a6b5231a18a5414dbd010d4d48f08dbd P 29806d3d37354c169484f6db368f8798 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } }
I20240502 01:54:45.260102 21844 ts_tablet_manager.cc:1398] T a6b5231a18a5414dbd010d4d48f08dbd P 29806d3d37354c169484f6db368f8798: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.000s
I20240502 01:54:45.261348 17520 tablet_copy_service.cc:342] P 311f765f59754bdcb4eadc04e14bd4ea: Request end of tablet copy session 29806d3d37354c169484f6db368f8798-a6b5231a18a5414dbd010d4d48f08dbd received from {username='slave'} at 127.9.176.197:51011
I20240502 01:54:45.261530 17520 tablet_copy_service.cc:433] P 311f765f59754bdcb4eadc04e14bd4ea: ending tablet copy session 29806d3d37354c169484f6db368f8798-a6b5231a18a5414dbd010d4d48f08dbd on tablet a6b5231a18a5414dbd010d4d48f08dbd with peer 29806d3d37354c169484f6db368f8798
I20240502 01:54:45.263990 22311 tablet_copy_client.cc:308] T fadcc8fe6ae047c1b142514aad8bebba P 4b7c382c9edd41babdd1a1afc904a813: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.193:35397
I20240502 01:54:45.265368 17520 tablet_copy_service.cc:140] P 311f765f59754bdcb4eadc04e14bd4ea: Received BeginTabletCopySession request for tablet fadcc8fe6ae047c1b142514aad8bebba from peer 4b7c382c9edd41babdd1a1afc904a813 ({username='slave'} at 127.9.176.198:53399)
I20240502 01:54:45.265620 17520 tablet_copy_service.cc:161] P 311f765f59754bdcb4eadc04e14bd4ea: Beginning new tablet copy session on tablet fadcc8fe6ae047c1b142514aad8bebba from peer 4b7c382c9edd41babdd1a1afc904a813 at {username='slave'} at 127.9.176.198:53399: session id = 4b7c382c9edd41babdd1a1afc904a813-fadcc8fe6ae047c1b142514aad8bebba
I20240502 01:54:45.273022 17520 tablet_copy_source_session.cc:205] T fadcc8fe6ae047c1b142514aad8bebba P 311f765f59754bdcb4eadc04e14bd4ea: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:45.277196 22311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fadcc8fe6ae047c1b142514aad8bebba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:45.323029 22311 tablet_copy_client.cc:783] T fadcc8fe6ae047c1b142514aad8bebba P 4b7c382c9edd41babdd1a1afc904a813: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:45.327605 22311 tablet_copy_client.cc:647] T fadcc8fe6ae047c1b142514aad8bebba P 4b7c382c9edd41babdd1a1afc904a813: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:45.341459 22311 tablet_copy_client.cc:523] T fadcc8fe6ae047c1b142514aad8bebba P 4b7c382c9edd41babdd1a1afc904a813: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:45.364967 22311 tablet_bootstrap.cc:492] T fadcc8fe6ae047c1b142514aad8bebba P 4b7c382c9edd41babdd1a1afc904a813: Bootstrap starting.
I20240502 01:54:45.393798 17897 consensus_queue.cc:237] T a5e03cf44eab41cc842342a138c04067 P da3584f429104d23a1a8899b35cc2138 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } }
W20240502 01:54:45.412073 17832 consensus_peers.cc:485] T a5e03cf44eab41cc842342a138c04067 P da3584f429104d23a1a8899b35cc2138 -> Peer 272b2179c1b4476090a89880df2a8baa (127.9.176.199:42269): Couldn't send request to peer 272b2179c1b4476090a89880df2a8baa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a5e03cf44eab41cc842342a138c04067. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:45.412642 17767 raft_consensus.cc:1261] T a5e03cf44eab41cc842342a138c04067 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Refusing update from remote peer da3584f429104d23a1a8899b35cc2138: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:45.412791 19467 raft_consensus.cc:1261] T a5e03cf44eab41cc842342a138c04067 P 29806d3d37354c169484f6db368f8798 [term 1 FOLLOWER]: Refusing update from remote peer da3584f429104d23a1a8899b35cc2138: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:45.413631 22261 consensus_queue.cc:1035] T a5e03cf44eab41cc842342a138c04067 P da3584f429104d23a1a8899b35cc2138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:45.414826 22261 consensus_queue.cc:1035] T a5e03cf44eab41cc842342a138c04067 P da3584f429104d23a1a8899b35cc2138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:45.432338 22276 raft_consensus.cc:2932] T a5e03cf44eab41cc842342a138c04067 P da3584f429104d23a1a8899b35cc2138 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 272b2179c1b4476090a89880df2a8baa (127.9.176.199) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } } }
I20240502 01:54:45.438048 17767 raft_consensus.cc:2932] T a5e03cf44eab41cc842342a138c04067 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 272b2179c1b4476090a89880df2a8baa (127.9.176.199) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } } }
I20240502 01:54:45.443899 17347 catalog_manager.cc:5484] T a5e03cf44eab41cc842342a138c04067 P da3584f429104d23a1a8899b35cc2138 reported cstate change: config changed from index 6 to 7, NON_VOTER 272b2179c1b4476090a89880df2a8baa (127.9.176.199) added. New cstate: current_term: 1 leader_uuid: "da3584f429104d23a1a8899b35cc2138" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:45.445358 21844 ts_tablet_manager.cc:902] T 38a60d7db47149ff90444e5276fc8935 P 29806d3d37354c169484f6db368f8798: Initiating tablet copy from peer 1b50a031f2784ba498b3e52e16916205 (127.9.176.194:45617)
I20240502 01:54:45.462929 22303 raft_consensus.cc:1050] T 88a137bcfc76410583f910b377d1be7f P 84d3a4b174c34e02aedab1e620731359: attempting to promote NON_VOTER 272b2179c1b4476090a89880df2a8baa to VOTER
I20240502 01:54:45.464004 22303 consensus_queue.cc:237] T 88a137bcfc76410583f910b377d1be7f P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } }
I20240502 01:54:45.465003 19467 raft_consensus.cc:2932] T a5e03cf44eab41cc842342a138c04067 P 29806d3d37354c169484f6db368f8798 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 272b2179c1b4476090a89880df2a8baa (127.9.176.199) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } } }
I20240502 01:54:45.474695 19603 raft_consensus.cc:1261] T 88a137bcfc76410583f910b377d1be7f P 4b7c382c9edd41babdd1a1afc904a813 [term 1 FOLLOWER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:45.475747 22305 consensus_queue.cc:1035] T 88a137bcfc76410583f910b377d1be7f P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:45.493799 21844 tablet_copy_client.cc:308] T 38a60d7db47149ff90444e5276fc8935 P 29806d3d37354c169484f6db368f8798: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.194:45617
I20240502 01:54:45.494879 17654 tablet_copy_service.cc:140] P 1b50a031f2784ba498b3e52e16916205: Received BeginTabletCopySession request for tablet 38a60d7db47149ff90444e5276fc8935 from peer 29806d3d37354c169484f6db368f8798 ({username='slave'} at 127.9.176.197:33725)
I20240502 01:54:45.495108 17654 tablet_copy_service.cc:161] P 1b50a031f2784ba498b3e52e16916205: Beginning new tablet copy session on tablet 38a60d7db47149ff90444e5276fc8935 from peer 29806d3d37354c169484f6db368f8798 at {username='slave'} at 127.9.176.197:33725: session id = 29806d3d37354c169484f6db368f8798-38a60d7db47149ff90444e5276fc8935
I20240502 01:54:45.496924 17500 raft_consensus.cc:1261] T 88a137bcfc76410583f910b377d1be7f P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:45.497748 22305 consensus_queue.cc:1035] T 88a137bcfc76410583f910b377d1be7f P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:45.500669 19736 raft_consensus.cc:1261] T 88a137bcfc76410583f910b377d1be7f P 272b2179c1b4476090a89880df2a8baa [term 1 LEARNER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:45.501542 22305 consensus_queue.cc:1035] T 88a137bcfc76410583f910b377d1be7f P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:45.502221 17654 tablet_copy_source_session.cc:205] T 38a60d7db47149ff90444e5276fc8935 P 1b50a031f2784ba498b3e52e16916205: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:45.509814 21844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38a60d7db47149ff90444e5276fc8935. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:45.512745 22305 raft_consensus.cc:2932] T 88a137bcfc76410583f910b377d1be7f P 84d3a4b174c34e02aedab1e620731359 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 272b2179c1b4476090a89880df2a8baa (127.9.176.199) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } }
I20240502 01:54:45.522584 19736 raft_consensus.cc:2932] T 88a137bcfc76410583f910b377d1be7f P 272b2179c1b4476090a89880df2a8baa [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 272b2179c1b4476090a89880df2a8baa (127.9.176.199) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } }
I20240502 01:54:45.529928 22311 tablet_bootstrap.cc:492] T fadcc8fe6ae047c1b142514aad8bebba P 4b7c382c9edd41babdd1a1afc904a813: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:45.531273 22311 tablet_bootstrap.cc:492] T fadcc8fe6ae047c1b142514aad8bebba P 4b7c382c9edd41babdd1a1afc904a813: Bootstrap complete.
I20240502 01:54:45.531630 22311 ts_tablet_manager.cc:1367] T fadcc8fe6ae047c1b142514aad8bebba P 4b7c382c9edd41babdd1a1afc904a813: Time spent bootstrapping tablet: real 0.168s	user 0.057s	sys 0.001s
I20240502 01:54:45.534129 22311 raft_consensus.cc:348] T fadcc8fe6ae047c1b142514aad8bebba P 4b7c382c9edd41babdd1a1afc904a813 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } }
I20240502 01:54:45.533998 17499 raft_consensus.cc:2932] T 88a137bcfc76410583f910b377d1be7f P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 272b2179c1b4476090a89880df2a8baa (127.9.176.199) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } }
I20240502 01:54:45.534510 22311 raft_consensus.cc:729] T fadcc8fe6ae047c1b142514aad8bebba P 4b7c382c9edd41babdd1a1afc904a813 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4b7c382c9edd41babdd1a1afc904a813, State: Initialized, Role: LEARNER
I20240502 01:54:45.534727 22311 consensus_queue.cc:260] T fadcc8fe6ae047c1b142514aad8bebba P 4b7c382c9edd41babdd1a1afc904a813 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } }
I20240502 01:54:45.535809 17347 catalog_manager.cc:5484] T 88a137bcfc76410583f910b377d1be7f P 272b2179c1b4476090a89880df2a8baa reported cstate change: config changed from index 7 to 8, 272b2179c1b4476090a89880df2a8baa (127.9.176.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "84d3a4b174c34e02aedab1e620731359" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } }
I20240502 01:54:45.540506 19603 raft_consensus.cc:2932] T 88a137bcfc76410583f910b377d1be7f P 4b7c382c9edd41babdd1a1afc904a813 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 272b2179c1b4476090a89880df2a8baa (127.9.176.199) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } }
I20240502 01:54:45.541384 22311 ts_tablet_manager.cc:1398] T fadcc8fe6ae047c1b142514aad8bebba P 4b7c382c9edd41babdd1a1afc904a813: Time spent starting tablet: real 0.009s	user 0.003s	sys 0.001s
I20240502 01:54:45.542152 17520 tablet_copy_service.cc:342] P 311f765f59754bdcb4eadc04e14bd4ea: Request end of tablet copy session 4b7c382c9edd41babdd1a1afc904a813-fadcc8fe6ae047c1b142514aad8bebba received from {username='slave'} at 127.9.176.198:53399
I20240502 01:54:45.542308 17520 tablet_copy_service.cc:433] P 311f765f59754bdcb4eadc04e14bd4ea: ending tablet copy session 4b7c382c9edd41babdd1a1afc904a813-fadcc8fe6ae047c1b142514aad8bebba on tablet fadcc8fe6ae047c1b142514aad8bebba with peer 4b7c382c9edd41babdd1a1afc904a813
I20240502 01:54:45.546772 17631 consensus_queue.cc:237] T df16e178e04040699201f325ce9e2979 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } }
I20240502 01:54:45.562808 19603 raft_consensus.cc:1203] T fadcc8fe6ae047c1b142514aad8bebba P 4b7c382c9edd41babdd1a1afc904a813 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 01:54:45.567900 19736 raft_consensus.cc:1261] T df16e178e04040699201f325ce9e2979 P 272b2179c1b4476090a89880df2a8baa [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:45.569007 22086 consensus_queue.cc:1035] T df16e178e04040699201f325ce9e2979 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:45.570511 21844 tablet_copy_client.cc:783] T 38a60d7db47149ff90444e5276fc8935 P 29806d3d37354c169484f6db368f8798: tablet copy: Starting download of 0 data blocks...
W20240502 01:54:45.576179 17565 consensus_peers.cc:485] T df16e178e04040699201f325ce9e2979 P 1b50a031f2784ba498b3e52e16916205 -> Peer 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198:36865): Couldn't send request to peer 4b7c382c9edd41babdd1a1afc904a813. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: df16e178e04040699201f325ce9e2979. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:45.582064 17767 consensus_queue.cc:237] T 88a137bcfc76410583f910b377d1be7f P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } }
I20240502 01:54:45.585342 17767 raft_consensus.cc:1261] T df16e178e04040699201f325ce9e2979 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:45.586272 21844 tablet_copy_client.cc:647] T 38a60d7db47149ff90444e5276fc8935 P 29806d3d37354c169484f6db368f8798: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:45.585286 19736 raft_consensus.cc:1261] T 88a137bcfc76410583f910b377d1be7f P 272b2179c1b4476090a89880df2a8baa [term 1 FOLLOWER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:54:45.589491 22305 consensus_queue.cc:1035] T 88a137bcfc76410583f910b377d1be7f P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:45.593087 22086 consensus_queue.cc:1035] T df16e178e04040699201f325ce9e2979 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:45.595001 21844 tablet_copy_client.cc:523] T 38a60d7db47149ff90444e5276fc8935 P 29806d3d37354c169484f6db368f8798: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:45.597415 19603 raft_consensus.cc:1261] T 88a137bcfc76410583f910b377d1be7f P 4b7c382c9edd41babdd1a1afc904a813 [term 1 FOLLOWER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:54:45.603919 22303 raft_consensus.cc:2932] T 88a137bcfc76410583f910b377d1be7f P 84d3a4b174c34e02aedab1e620731359 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } }
I20240502 01:54:45.605492 22300 consensus_queue.cc:1035] T 88a137bcfc76410583f910b377d1be7f P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:45.608810 19604 raft_consensus.cc:2932] T 88a137bcfc76410583f910b377d1be7f P 4b7c382c9edd41babdd1a1afc904a813 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } }
I20240502 01:54:45.612694 17343 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 88a137bcfc76410583f910b377d1be7f with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:45.613909 19736 raft_consensus.cc:2932] T 88a137bcfc76410583f910b377d1be7f P 272b2179c1b4476090a89880df2a8baa [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } }
I20240502 01:54:45.617825 17631 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3b3fa060e8d64911bec810ec07a358d2"
dest_uuid: "1b50a031f2784ba498b3e52e16916205"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:55538
I20240502 01:54:45.618232 17631 raft_consensus.cc:595] T 3b3fa060e8d64911bec810ec07a358d2 P 1b50a031f2784ba498b3e52e16916205 [term 1 LEADER]: Received request to transfer leadership
I20240502 01:54:45.618861 17351 catalog_manager.cc:5484] T 88a137bcfc76410583f910b377d1be7f P 4b7c382c9edd41babdd1a1afc904a813 reported cstate change: config changed from index 8 to 9, VOTER 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193) evicted. New cstate: current_term: 1 leader_uuid: "84d3a4b174c34e02aedab1e620731359" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } }
I20240502 01:54:45.621102 21844 tablet_bootstrap.cc:492] T 38a60d7db47149ff90444e5276fc8935 P 29806d3d37354c169484f6db368f8798: Bootstrap starting.
I20240502 01:54:45.620704 21926 raft_consensus.cc:2932] T df16e178e04040699201f325ce9e2979 P 1b50a031f2784ba498b3e52e16916205 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } } }
I20240502 01:54:45.624519 19738 raft_consensus.cc:2932] T df16e178e04040699201f325ce9e2979 P 272b2179c1b4476090a89880df2a8baa [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } } }
I20240502 01:54:45.625787 17478 tablet_service.cc:1514] Processing DeleteTablet for tablet 88a137bcfc76410583f910b377d1be7f with delete_type TABLET_DATA_TOMBSTONED (TS 311f765f59754bdcb4eadc04e14bd4ea not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:48974
I20240502 01:54:45.626538 22375 tablet_replica.cc:330] T 88a137bcfc76410583f910b377d1be7f P 311f765f59754bdcb4eadc04e14bd4ea: stopping tablet replica
I20240502 01:54:45.627097 22375 raft_consensus.cc:2229] T 88a137bcfc76410583f910b377d1be7f P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:45.628059 22375 raft_consensus.cc:2258] T 88a137bcfc76410583f910b377d1be7f P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:45.635358 17347 catalog_manager.cc:5484] T df16e178e04040699201f325ce9e2979 P 1b50a031f2784ba498b3e52e16916205 reported cstate change: config changed from index 6 to 7, NON_VOTER 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) added. New cstate: current_term: 1 leader_uuid: "1b50a031f2784ba498b3e52e16916205" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:45.639351 17631 consensus_queue.cc:237] T 91e0e82526c0489a8908e250bf3c851f P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: NON_VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { promote: true } }
I20240502 01:54:45.643270 19872 raft_consensus.cc:1261] T 91e0e82526c0489a8908e250bf3c851f P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:54:45.644304 21926 consensus_queue.cc:1035] T 91e0e82526c0489a8908e250bf3c851f P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:54:45.648258 19467 raft_consensus.cc:1261] T 91e0e82526c0489a8908e250bf3c851f P 29806d3d37354c169484f6db368f8798 [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:54:45.649518 22290 consensus_queue.cc:1035] T 91e0e82526c0489a8908e250bf3c851f P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:54:45.650974 22290 raft_consensus.cc:2932] T 91e0e82526c0489a8908e250bf3c851f P 1b50a031f2784ba498b3e52e16916205 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: NON_VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { promote: true } } }
I20240502 01:54:45.658669 19872 raft_consensus.cc:2932] T 91e0e82526c0489a8908e250bf3c851f P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: NON_VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { promote: true } } }
I20240502 01:54:45.669744 17351 catalog_manager.cc:5484] T 91e0e82526c0489a8908e250bf3c851f P 94931f4b51c9493cb3d4264ce99f5e76 reported cstate change: config changed from index 9 to 10, NON_VOTER 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193) added. New cstate: current_term: 1 leader_uuid: "1b50a031f2784ba498b3e52e16916205" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: NON_VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { promote: true } } }
I20240502 01:54:45.686700 22290 raft_consensus.cc:979] T 3b3fa060e8d64911bec810ec07a358d2 P 1b50a031f2784ba498b3e52e16916205: : Instructing follower da3584f429104d23a1a8899b35cc2138 to start an election
I20240502 01:54:45.686972 22290 raft_consensus.cc:1067] T 3b3fa060e8d64911bec810ec07a358d2 P 1b50a031f2784ba498b3e52e16916205 [term 1 LEADER]: Signalling peer da3584f429104d23a1a8899b35cc2138 to start an election
I20240502 01:54:45.688534 17897 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3b3fa060e8d64911bec810ec07a358d2"
dest_uuid: "da3584f429104d23a1a8899b35cc2138"
 from {username='slave'} at 127.9.176.194:46577
I20240502 01:54:45.689018 17897 raft_consensus.cc:482] T 3b3fa060e8d64911bec810ec07a358d2 P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:45.689226 17897 raft_consensus.cc:3037] T 3b3fa060e8d64911bec810ec07a358d2 P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:45.694031 17897 raft_consensus.cc:504] T 3b3fa060e8d64911bec810ec07a358d2 P da3584f429104d23a1a8899b35cc2138 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } }
I20240502 01:54:45.696703 17631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b3fa060e8d64911bec810ec07a358d2" candidate_uuid: "da3584f429104d23a1a8899b35cc2138" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: true dest_uuid: "1b50a031f2784ba498b3e52e16916205"
I20240502 01:54:45.697093 19604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b3fa060e8d64911bec810ec07a358d2" candidate_uuid: "da3584f429104d23a1a8899b35cc2138" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: true dest_uuid: "4b7c382c9edd41babdd1a1afc904a813"
I20240502 01:54:45.697593 19604 raft_consensus.cc:3037] T 3b3fa060e8d64911bec810ec07a358d2 P 4b7c382c9edd41babdd1a1afc904a813 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:45.703543 19872 raft_consensus.cc:1203] T 62fd543d5c7c41f98926f2148ee61e7d P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240502 01:54:45.705565 19604 raft_consensus.cc:2454] T 3b3fa060e8d64911bec810ec07a358d2 P 4b7c382c9edd41babdd1a1afc904a813 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da3584f429104d23a1a8899b35cc2138 in term 2.
I20240502 01:54:45.706543 17897 leader_election.cc:288] T 3b3fa060e8d64911bec810ec07a358d2 P da3584f429104d23a1a8899b35cc2138 [CANDIDATE]: Term 2 election: Requested vote from peers 1b50a031f2784ba498b3e52e16916205 (127.9.176.194:45617), 272b2179c1b4476090a89880df2a8baa (127.9.176.199:42269), 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198:36865)
W20240502 01:54:45.710366 17568 consensus_peers.cc:485] T 91e0e82526c0489a8908e250bf3c851f P 1b50a031f2784ba498b3e52e16916205 -> Peer 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193:35397): Couldn't send request to peer 311f765f59754bdcb4eadc04e14bd4ea. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:45.711953 19736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b3fa060e8d64911bec810ec07a358d2" candidate_uuid: "da3584f429104d23a1a8899b35cc2138" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: true dest_uuid: "272b2179c1b4476090a89880df2a8baa"
I20240502 01:54:45.712491 19736 raft_consensus.cc:3037] T 3b3fa060e8d64911bec810ec07a358d2 P 272b2179c1b4476090a89880df2a8baa [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:45.714203 22375 ts_tablet_manager.cc:1875] T 88a137bcfc76410583f910b377d1be7f P 311f765f59754bdcb4eadc04e14bd4ea: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:45.716992 19467 raft_consensus.cc:2932] T 91e0e82526c0489a8908e250bf3c851f P 29806d3d37354c169484f6db368f8798 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: NON_VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { promote: true } } }
I20240502 01:54:45.711373 17631 raft_consensus.cc:3032] T 3b3fa060e8d64911bec810ec07a358d2 P 1b50a031f2784ba498b3e52e16916205 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:54:45.720487 17631 raft_consensus.cc:729] T 3b3fa060e8d64911bec810ec07a358d2 P 1b50a031f2784ba498b3e52e16916205 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1b50a031f2784ba498b3e52e16916205, State: Running, Role: LEADER
I20240502 01:54:45.720958 17631 consensus_queue.cc:260] T 3b3fa060e8d64911bec810ec07a358d2 P 1b50a031f2784ba498b3e52e16916205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } }
I20240502 01:54:45.721532 17631 raft_consensus.cc:3037] T 3b3fa060e8d64911bec810ec07a358d2 P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:45.722707 22375 ts_tablet_manager.cc:1888] T 88a137bcfc76410583f910b377d1be7f P 311f765f59754bdcb4eadc04e14bd4ea: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 01:54:45.723331 22375 log.cc:1196] T 88a137bcfc76410583f910b377d1be7f P 311f765f59754bdcb4eadc04e14bd4ea: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/wal/wals/88a137bcfc76410583f910b377d1be7f
I20240502 01:54:45.724625 17343 catalog_manager.cc:4830] TS 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193:35397): tablet 88a137bcfc76410583f910b377d1be7f (table rebalance_test_table_3 [id=c5e1575e337a4256a90ce5b685b3ff28]) successfully deleted
I20240502 01:54:45.725076 17631 raft_consensus.cc:2454] T 3b3fa060e8d64911bec810ec07a358d2 P 1b50a031f2784ba498b3e52e16916205 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da3584f429104d23a1a8899b35cc2138 in term 2.
I20240502 01:54:45.726051 17835 leader_election.cc:302] T 3b3fa060e8d64911bec810ec07a358d2 P da3584f429104d23a1a8899b35cc2138 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1b50a031f2784ba498b3e52e16916205, 4b7c382c9edd41babdd1a1afc904a813, da3584f429104d23a1a8899b35cc2138; no voters: 
I20240502 01:54:45.726501 22350 raft_consensus.cc:2781] T 3b3fa060e8d64911bec810ec07a358d2 P da3584f429104d23a1a8899b35cc2138 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:54:45.726687 22350 raft_consensus.cc:686] T 3b3fa060e8d64911bec810ec07a358d2 P da3584f429104d23a1a8899b35cc2138 [term 2 LEADER]: Becoming Leader. State: Replica: da3584f429104d23a1a8899b35cc2138, State: Running, Role: LEADER
I20240502 01:54:45.727058 22350 consensus_queue.cc:237] T 3b3fa060e8d64911bec810ec07a358d2 P da3584f429104d23a1a8899b35cc2138 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } }
I20240502 01:54:45.730855 17764 raft_consensus.cc:2932] T df16e178e04040699201f325ce9e2979 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: true } } }
I20240502 01:54:45.732374 19736 raft_consensus.cc:2454] T 3b3fa060e8d64911bec810ec07a358d2 P 272b2179c1b4476090a89880df2a8baa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da3584f429104d23a1a8899b35cc2138 in term 2.
I20240502 01:54:45.737027 17352 catalog_manager.cc:5484] T 3b3fa060e8d64911bec810ec07a358d2 P da3584f429104d23a1a8899b35cc2138 reported cstate change: term changed from 1 to 2, leader changed from 1b50a031f2784ba498b3e52e16916205 (127.9.176.194) to da3584f429104d23a1a8899b35cc2138 (127.9.176.196). New cstate: current_term: 2 leader_uuid: "da3584f429104d23a1a8899b35cc2138" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:45.762537 17897 consensus_queue.cc:237] T 17365abbb6c1430aa278719eb8e4a648 P da3584f429104d23a1a8899b35cc2138 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false replace: true } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } }
I20240502 01:54:45.776157 17767 consensus_queue.cc:237] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } }
I20240502 01:54:45.776892 19464 raft_consensus.cc:1203] T a6b5231a18a5414dbd010d4d48f08dbd P 29806d3d37354c169484f6db368f8798 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
W20240502 01:54:45.780094 17832 consensus_peers.cc:485] T 17365abbb6c1430aa278719eb8e4a648 P da3584f429104d23a1a8899b35cc2138 -> Peer 29806d3d37354c169484f6db368f8798 (127.9.176.197:35011): Couldn't send request to peer 29806d3d37354c169484f6db368f8798. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 17365abbb6c1430aa278719eb8e4a648. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:45.781065 19604 raft_consensus.cc:1261] T 17365abbb6c1430aa278719eb8e4a648 P 4b7c382c9edd41babdd1a1afc904a813 [term 1 FOLLOWER]: Refusing update from remote peer da3584f429104d23a1a8899b35cc2138: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:45.782099 22350 consensus_queue.cc:1035] T 17365abbb6c1430aa278719eb8e4a648 P da3584f429104d23a1a8899b35cc2138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:45.783818 17500 raft_consensus.cc:1261] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:45.784535 22303 consensus_queue.cc:1035] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 01:54:45.785889 17700 consensus_peers.cc:485] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 84d3a4b174c34e02aedab1e620731359 -> Peer 272b2179c1b4476090a89880df2a8baa (127.9.176.199:42269): Couldn't send request to peer 272b2179c1b4476090a89880df2a8baa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3ca5c4988ff74e3ca8d7a5614bcb93e2. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:45.791267 17764 raft_consensus.cc:1261] T 17365abbb6c1430aa278719eb8e4a648 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Refusing update from remote peer da3584f429104d23a1a8899b35cc2138: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:45.792289 19467 raft_consensus.cc:1261] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 29806d3d37354c169484f6db368f8798 [term 1 FOLLOWER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:45.792272 22261 consensus_queue.cc:1035] T 17365abbb6c1430aa278719eb8e4a648 P da3584f429104d23a1a8899b35cc2138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:45.792135 17631 consensus_queue.cc:237] T b40f968655774ce887dff8b7f4651501 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } }
I20240502 01:54:45.793015 22305 consensus_queue.cc:1035] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 01:54:45.804803 17565 consensus_peers.cc:485] T b40f968655774ce887dff8b7f4651501 P 1b50a031f2784ba498b3e52e16916205 -> Peer 29806d3d37354c169484f6db368f8798 (127.9.176.197:35011): Couldn't send request to peer 29806d3d37354c169484f6db368f8798. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b40f968655774ce887dff8b7f4651501. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:45.806437 19603 raft_consensus.cc:1261] T b40f968655774ce887dff8b7f4651501 P 4b7c382c9edd41babdd1a1afc904a813 [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:54:45.807214 22290 consensus_queue.cc:1035] T b40f968655774ce887dff8b7f4651501 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:54:45.811758 19736 raft_consensus.cc:1261] T b40f968655774ce887dff8b7f4651501 P 272b2179c1b4476090a89880df2a8baa [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:54:45.812680 22290 consensus_queue.cc:1035] T b40f968655774ce887dff8b7f4651501 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:54:45.812765 22301 raft_consensus.cc:2932] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 84d3a4b174c34e02aedab1e620731359 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 272b2179c1b4476090a89880df2a8baa (127.9.176.199) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } } }
I20240502 01:54:45.813632 19467 raft_consensus.cc:2932] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 29806d3d37354c169484f6db368f8798 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 272b2179c1b4476090a89880df2a8baa (127.9.176.199) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } } }
I20240502 01:54:45.821878 22261 raft_consensus.cc:2932] T 17365abbb6c1430aa278719eb8e4a648 P da3584f429104d23a1a8899b35cc2138 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 29806d3d37354c169484f6db368f8798 (127.9.176.197) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false replace: true } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } } }
I20240502 01:54:45.822541 17352 catalog_manager.cc:5484] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 84d3a4b174c34e02aedab1e620731359 reported cstate change: config changed from index 6 to 7, NON_VOTER 272b2179c1b4476090a89880df2a8baa (127.9.176.199) added. New cstate: current_term: 1 leader_uuid: "84d3a4b174c34e02aedab1e620731359" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:45.825912 19602 raft_consensus.cc:2932] T 17365abbb6c1430aa278719eb8e4a648 P 4b7c382c9edd41babdd1a1afc904a813 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 29806d3d37354c169484f6db368f8798 (127.9.176.197) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false replace: true } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } } }
I20240502 01:54:45.836536 17352 catalog_manager.cc:5484] T 17365abbb6c1430aa278719eb8e4a648 P da3584f429104d23a1a8899b35cc2138 reported cstate change: config changed from index 6 to 7, NON_VOTER 29806d3d37354c169484f6db368f8798 (127.9.176.197) added. New cstate: current_term: 1 leader_uuid: "da3584f429104d23a1a8899b35cc2138" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:45.842823 17764 raft_consensus.cc:2932] T 17365abbb6c1430aa278719eb8e4a648 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 29806d3d37354c169484f6db368f8798 (127.9.176.197) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false replace: true } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } } }
I20240502 01:54:45.844830 17499 raft_consensus.cc:2932] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 272b2179c1b4476090a89880df2a8baa (127.9.176.199) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } } }
I20240502 01:54:45.861271 22290 raft_consensus.cc:2932] T b40f968655774ce887dff8b7f4651501 P 1b50a031f2784ba498b3e52e16916205 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 29806d3d37354c169484f6db368f8798 (127.9.176.197) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } } }
I20240502 01:54:45.864513 19738 raft_consensus.cc:2932] T b40f968655774ce887dff8b7f4651501 P 272b2179c1b4476090a89880df2a8baa [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 29806d3d37354c169484f6db368f8798 (127.9.176.197) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } } }
I20240502 01:54:45.879073 17350 catalog_manager.cc:5484] T b40f968655774ce887dff8b7f4651501 P 272b2179c1b4476090a89880df2a8baa reported cstate change: config changed from index 9 to 10, NON_VOTER 29806d3d37354c169484f6db368f8798 (127.9.176.197) added. New cstate: current_term: 1 leader_uuid: "1b50a031f2784ba498b3e52e16916205" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } } }
I20240502 01:54:45.919106 19604 raft_consensus.cc:2932] T b40f968655774ce887dff8b7f4651501 P 4b7c382c9edd41babdd1a1afc904a813 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 29806d3d37354c169484f6db368f8798 (127.9.176.197) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } } }
I20240502 01:54:45.966591 22441 ts_tablet_manager.cc:902] T a5e03cf44eab41cc842342a138c04067 P 272b2179c1b4476090a89880df2a8baa: Initiating tablet copy from peer da3584f429104d23a1a8899b35cc2138 (127.9.176.196:46381)
I20240502 01:54:45.989027 21844 tablet_bootstrap.cc:492] T 38a60d7db47149ff90444e5276fc8935 P 29806d3d37354c169484f6db368f8798: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:45.989631 21844 tablet_bootstrap.cc:492] T 38a60d7db47149ff90444e5276fc8935 P 29806d3d37354c169484f6db368f8798: Bootstrap complete.
I20240502 01:54:45.989985 21844 ts_tablet_manager.cc:1367] T 38a60d7db47149ff90444e5276fc8935 P 29806d3d37354c169484f6db368f8798: Time spent bootstrapping tablet: real 0.369s	user 0.053s	sys 0.008s
I20240502 01:54:45.990952 21844 raft_consensus.cc:348] T 38a60d7db47149ff90444e5276fc8935 P 29806d3d37354c169484f6db368f8798 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } }
I20240502 01:54:45.991264 21844 raft_consensus.cc:729] T 38a60d7db47149ff90444e5276fc8935 P 29806d3d37354c169484f6db368f8798 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 29806d3d37354c169484f6db368f8798, State: Initialized, Role: LEARNER
I20240502 01:54:45.991513 21844 consensus_queue.cc:260] T 38a60d7db47149ff90444e5276fc8935 P 29806d3d37354c169484f6db368f8798 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: true } }
I20240502 01:54:45.992453 21844 ts_tablet_manager.cc:1398] T 38a60d7db47149ff90444e5276fc8935 P 29806d3d37354c169484f6db368f8798: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.000s
I20240502 01:54:45.993844 17654 tablet_copy_service.cc:342] P 1b50a031f2784ba498b3e52e16916205: Request end of tablet copy session 29806d3d37354c169484f6db368f8798-38a60d7db47149ff90444e5276fc8935 received from {username='slave'} at 127.9.176.197:33725
I20240502 01:54:45.994027 17654 tablet_copy_service.cc:433] P 1b50a031f2784ba498b3e52e16916205: ending tablet copy session 29806d3d37354c169484f6db368f8798-38a60d7db47149ff90444e5276fc8935 on tablet 38a60d7db47149ff90444e5276fc8935 with peer 29806d3d37354c169484f6db368f8798
I20240502 01:54:46.033955 22311 ts_tablet_manager.cc:902] T df16e178e04040699201f325ce9e2979 P 4b7c382c9edd41babdd1a1afc904a813: Initiating tablet copy from peer 1b50a031f2784ba498b3e52e16916205 (127.9.176.194:45617)
F20240502 01:54:46.034397 22311 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-05b67d9912f047bfb6a2bed90b82ae15.queue_time_us, tablet-download-pool-05b67d9912f047bfb6a2bed90b82ae15.queue_time_us) (tablet-download-pool-05b67d9912f047bfb6a2bed90b82ae15.run_cpu_time_us, tablet-download-pool-05b67d9912f047bfb6a2bed90b82ae15.run_cpu_time_us) (tablet-download-pool-05b67d9912f047bfb6a2bed90b82ae15.run_wall_time_us, tablet-download-pool-05b67d9912f047bfb6a2bed90b82ae15.run_wall_time_us) (tablet-download-pool-17365abbb6c1430aa278719eb8e4a648.queue_time_us, tablet-download-pool-17365abbb6c1430aa278719eb8e4a648.queue_time_us) (tablet-download-pool-17365abbb6c1430aa278719eb8e4a648.run_cpu_time_us, tablet-download-pool-17365abbb6c1430aa278719eb8e4a648.run_cpu_time_us) (tablet-download-pool-17365abbb6c1430aa278719eb8e4a648.run_wall_time_us, tablet-download-pool-17365abbb6c1430aa278719eb8e4a648.run_wall_time_us) (tablet-download-pool-37eda67598fe491bbb29aa62b2c4bec7.queue_time_us, tablet-download-pool-37eda67598fe491bbb29aa62b2c4bec7.queue_time_us) (tablet-download-pool-37eda67598fe491bbb29aa62b2c4bec7.run_cpu_time_us, tablet-download-pool-37eda67598fe491bbb29aa62b2c4bec7.run_cpu_time_us) (tablet-download-pool-37eda67598fe491bbb29aa62b2c4bec7.run_wall_time_us, tablet-download-pool-37eda67598fe491bbb29aa62b2c4bec7.run_wall_time_us) (tablet-download-pool-3b3fa060e8d64911bec810ec07a358d2.queue_time_us, tablet-download-pool-3b3fa060e8d64911bec810ec07a358d2.queue_time_us) (tablet-download-pool-3b3fa060e8d64911bec810ec07a358d2.run_cpu_time_us, tablet-download-pool-3b3fa060e8d64911bec810ec07a358d2.run_cpu_time_us) (tablet-download-pool-3b3fa060e8d64911bec810ec07a358d2.run_wall_time_us, tablet-download-pool-3b3fa060e8d64911bec810ec07a358d2.run_wall_time_us) (tablet-download-pool-59e507855e5043d38cef84e92adf437b.queue_time_us, tablet-download-pool-59e507855e5043d38cef84e92adf437b.queue_time_us) (tablet-download-pool-59e507855e5043d38cef84e92adf437b.run_cpu_time_us, tablet-download-pool-59e507855e5043d38cef84e92adf437b.run_cpu_time_us) (tablet-download-pool-59e507855e5043d38cef84e92adf437b.run_wall_time_us, tablet-download-pool-59e507855e5043d38cef84e92adf437b.run_wall_time_us) (tablet-download-pool-5f3f3e5f772b4566b69236fd87d71a38.queue_time_us, tablet-download-pool-5f3f3e5f772b4566b69236fd87d71a38.queue_time_us) (tablet-download-pool-5f3f3e5f772b4566b69236fd87d71a38.run_cpu_time_us, tablet-download-pool-5f3f3e5f772b4566b69236fd87d71a38.run_cpu_time_us) (tablet-download-pool-5f3f3e5f772b4566b69236fd87d71a38.run_wall_time_us, tablet-download-pool-5f3f3e5f772b4566b69236fd87d71a38.run_wall_time_us) (tablet-download-pool-768e7cbf63a94d3a815ac77a4d0acf41.queue_time_us, tablet-download-pool-768e7cbf63a94d3a815ac77a4d0acf41.queue_time_us) (tablet-download-pool-768e7cbf63a94d3a815ac77a4d0acf41.run_cpu_time_us, tablet-download-pool-768e7cbf63a94d3a815ac77a4d0acf41.run_cpu_time_us) (tablet-download-pool-768e7cbf63a94d3a815ac77a4d0acf41.run_wall_time_us, tablet-download-pool-768e7cbf63a94d3a815ac77a4d0acf41.run_wall_time_us) (tablet-download-pool-7ce9d4281c1b46258493feb39e8d9ab2.queue_time_us, tablet-download-pool-7ce9d4281c1b46258493feb39e8d9ab2.queue_time_us) (tablet-download-pool-7ce9d4281c1b46258493feb39e8d9ab2.run_cpu_time_us, tablet-download-pool-7ce9d4281c1b46258493feb39e8d9ab2.run_cpu_time_us) (tablet-download-pool-7ce9d4281c1b46258493feb39e8d9ab2.run_wall_time_us, tablet-download-pool-7ce9d4281c1b46258493feb39e8d9ab2.run_wall_time_us) (tablet-download-pool-88a137bcfc76410583f910b377d1be7f.queue_time_us, tablet-download-pool-88a137bcfc76410583f910b377d1be7f.queue_time_us) (tablet-download-pool-88a137bcfc76410583f910b377d1be7f.run_cpu_time_us, tablet-download-pool-88a137bcfc76410583f910b377d1be7f.run_cpu_time_us) (tablet-download-pool-88a137bcfc76410583f910b377d1be7f.run_wall_time_us, tablet-download-pool-88a137bcfc76410583f910b377d1be7f.run_wall_time_us) (tablet-download-pool-a6b5231a18a5414dbd010d4d48f08dbd.queue_time_us, tablet-download-pool-a6b5231a18a5414dbd010d4d48f08dbd.queue_time_us) (tablet-download-pool-a6b5231a18a5414dbd010d4d48f08dbd.run_cpu_time_us, tablet-download-pool-a6b5231a18a5414dbd010d4d48f08dbd.run_cpu_time_us) (tablet-download-pool-a6b5231a18a5414dbd010d4d48f08dbd.run_wall_time_us, tablet-download-pool-a6b5231a18a5414dbd010d4d48f08dbd.run_wall_time_us) (tablet-download-pool-b40f968655774ce887dff8b7f4651501.queue_time_us, tablet-download-pool-b40f968655774ce887dff8b7f4651501.queue_time_us) (tablet-download-pool-b40f968655774ce887dff8b7f4651501.run_cpu_time_us, tablet-download-pool-b40f968655774ce887dff8b7f4651501.run_cpu_time_us) (tablet-download-pool-b40f968655774ce887dff8b7f4651501.run_wall_time_us, tablet-download-pool-b40f968655774ce887dff8b7f4651501.run_wall_time_us) (tablet-download-pool-c91f74f46b04425aa5393515cb9f781c.queue_time_us, tablet-download-pool-c91f74f46b04425aa5393515cb9f781c.queue_time_us) (tablet-download-pool-c91f74f46b04425aa5393515cb9f781c.run_cpu_time_us, tablet-download-pool-c91f74f46b04425aa5393515cb9f781c.run_cpu_time_us) (tablet-download-pool-c91f74f46b04425aa5393515cb9f781c.run_wall_time_us, tablet-download-pool-c91f74f46b04425aa5393515cb9f781c.run_wall_time_us) (tablet-download-pool-df16e178e04040699201f325ce9e2979.queue_time_us, tablet-download-pool-df16e178e04040699201f325ce9e2979.queue_time_us) (tablet-download-pool-f07e9b433e554c85b3c25b7e708f8b2f.queue_time_us, tablet-download-pool-f07e9b433e554c85b3c25b7e708f8b2f.queue_time_us) (tablet-download-pool-f07e9b433e554c85b3c25b7e708f8b2f.run_cpu_time_us, tablet-download-pool-f07e9b433e554c85b3c25b7e708f8b2f.run_cpu_time_us) (tablet-download-pool-f07e9b433e554c85b3c25b7e708f8b2f.run_wall_time_us, tablet-download-pool-f07e9b433e554c85b3c25b7e708f8b2f.run_wall_time_us) (tablet-download-pool-fadcc8fe6ae047c1b142514aad8bebba.queue_time_us, tablet-download-pool-fadcc8fe6ae047c1b142514aad8bebba.queue_time_us) (tablet-download-pool-fadcc8fe6ae047c1b142514aad8bebba.run_cpu_time_us, tablet-download-pool-fadcc8fe6ae047c1b142514aad8bebba.run_cpu_time_us) (tablet-download-pool-fadcc8fe6ae047c1b142514aad8bebba.run_wall_time_us, tablet-download-pool-fadcc8fe6ae047c1b142514aad8bebba.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
    @     0x7fa471d97dcd  google::LogMessage::Fail() at ??:0
    @     0x7fa471d9bb93  google::LogMessage::SendToLog() at ??:0
    @     0x7fa471d977cc  google::LogMessage::Flush() at ??:0
    @     0x7fa471d98f59  google::LogMessageFatal::~LogMessageFatal() at ??:0
I20240502 01:54:46.057426 22441 tablet_copy_client.cc:308] T a5e03cf44eab41cc842342a138c04067 P 272b2179c1b4476090a89880df2a8baa: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.196:46381
I20240502 01:54:46.059207 17922 tablet_copy_service.cc:140] P da3584f429104d23a1a8899b35cc2138: Received BeginTabletCopySession request for tablet a5e03cf44eab41cc842342a138c04067 from peer 272b2179c1b4476090a89880df2a8baa ({username='slave'} at 127.9.176.199:46107)
I20240502 01:54:46.059476 17922 tablet_copy_service.cc:161] P da3584f429104d23a1a8899b35cc2138: Beginning new tablet copy session on tablet a5e03cf44eab41cc842342a138c04067 from peer 272b2179c1b4476090a89880df2a8baa at {username='slave'} at 127.9.176.199:46107: session id = 272b2179c1b4476090a89880df2a8baa-a5e03cf44eab41cc842342a138c04067
I20240502 01:54:46.064846 17922 tablet_copy_source_session.cc:205] T a5e03cf44eab41cc842342a138c04067 P da3584f429104d23a1a8899b35cc2138: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:46.067145 22441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5e03cf44eab41cc842342a138c04067. 1 dirs total, 0 dirs full, 0 dirs failed
    @     0x7fa474ad8742  kudu::TraceMetrics::InternName() at ??:0
I20240502 01:54:46.076351 22441 tablet_copy_client.cc:783] T a5e03cf44eab41cc842342a138c04067 P 272b2179c1b4476090a89880df2a8baa: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:46.076664 22441 tablet_copy_client.cc:647] T a5e03cf44eab41cc842342a138c04067 P 272b2179c1b4476090a89880df2a8baa: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:46.085268 22206 raft_consensus.cc:1050] T fadcc8fe6ae047c1b142514aad8bebba P 311f765f59754bdcb4eadc04e14bd4ea: attempting to promote NON_VOTER 4b7c382c9edd41babdd1a1afc904a813 to VOTER
I20240502 01:54:46.086407 22206 consensus_queue.cc:237] T fadcc8fe6ae047c1b142514aad8bebba P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } }
I20240502 01:54:46.092918 22441 tablet_copy_client.cc:523] T a5e03cf44eab41cc842342a138c04067 P 272b2179c1b4476090a89880df2a8baa: tablet copy: Tablet Copy complete. Replacing tablet superblock.
    @     0x7fa474aa4f8b  kudu::ThreadPool::ThreadPool() at ??:0
I20240502 01:54:46.108587 19604 raft_consensus.cc:1261] T fadcc8fe6ae047c1b142514aad8bebba P 4b7c382c9edd41babdd1a1afc904a813 [term 1 LEARNER]: Refusing update from remote peer 311f765f59754bdcb4eadc04e14bd4ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:46.109440 22197 consensus_queue.cc:1035] T fadcc8fe6ae047c1b142514aad8bebba P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
    @     0x7fa474a9ecd6  kudu::ThreadPoolBuilder::Build() at ??:0
    @     0x7fa483ef18de  kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
I20240502 01:54:46.139523 22441 tablet_bootstrap.cc:492] T a5e03cf44eab41cc842342a138c04067 P 272b2179c1b4476090a89880df2a8baa: Bootstrap starting.
I20240502 01:54:46.144778 22487 ts_tablet_manager.cc:902] T 91e0e82526c0489a8908e250bf3c851f P 311f765f59754bdcb4eadc04e14bd4ea: Initiating tablet copy from peer 1b50a031f2784ba498b3e52e16916205 (127.9.176.194:45617)
I20240502 01:54:46.153216 17764 raft_consensus.cc:1261] T fadcc8fe6ae047c1b142514aad8bebba P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Refusing update from remote peer 311f765f59754bdcb4eadc04e14bd4ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:46.153880 17897 raft_consensus.cc:1261] T fadcc8fe6ae047c1b142514aad8bebba P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Refusing update from remote peer 311f765f59754bdcb4eadc04e14bd4ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:46.155050 22482 consensus_queue.cc:1035] T fadcc8fe6ae047c1b142514aad8bebba P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:46.156507 22481 consensus_queue.cc:1035] T fadcc8fe6ae047c1b142514aad8bebba P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:46.162366 21844 ts_tablet_manager.cc:902] T 17365abbb6c1430aa278719eb8e4a648 P 29806d3d37354c169484f6db368f8798: Initiating tablet copy from peer da3584f429104d23a1a8899b35cc2138 (127.9.176.196:46381)
    @     0x7fa483f09daf  kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
I20240502 01:54:46.189606 22487 tablet_copy_client.cc:308] T 91e0e82526c0489a8908e250bf3c851f P 311f765f59754bdcb4eadc04e14bd4ea: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.194:45617
I20240502 01:54:46.192898 19604 raft_consensus.cc:1261] T 3b3fa060e8d64911bec810ec07a358d2 P 4b7c382c9edd41babdd1a1afc904a813 [term 2 FOLLOWER]: Refusing update from remote peer da3584f429104d23a1a8899b35cc2138: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240502 01:54:46.193835 22261 consensus_queue.cc:1035] T 3b3fa060e8d64911bec810ec07a358d2 P da3584f429104d23a1a8899b35cc2138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 01:54:46.194190 22197 raft_consensus.cc:1050] T 62fd543d5c7c41f98926f2148ee61e7d P 311f765f59754bdcb4eadc04e14bd4ea: attempting to promote NON_VOTER 94931f4b51c9493cb3d4264ce99f5e76 to VOTER
I20240502 01:54:46.194545 17654 tablet_copy_service.cc:140] P 1b50a031f2784ba498b3e52e16916205: Received BeginTabletCopySession request for tablet 91e0e82526c0489a8908e250bf3c851f from peer 311f765f59754bdcb4eadc04e14bd4ea ({username='slave'} at 127.9.176.193:52149)
I20240502 01:54:46.194741 17654 tablet_copy_service.cc:161] P 1b50a031f2784ba498b3e52e16916205: Beginning new tablet copy session on tablet 91e0e82526c0489a8908e250bf3c851f from peer 311f765f59754bdcb4eadc04e14bd4ea at {username='slave'} at 127.9.176.193:52149: session id = 311f765f59754bdcb4eadc04e14bd4ea-91e0e82526c0489a8908e250bf3c851f
I20240502 01:54:46.195149 22197 consensus_queue.cc:237] T 62fd543d5c7c41f98926f2148ee61e7d P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } }
    @     0x7fa484034480  kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
I20240502 01:54:46.198925 17654 tablet_copy_source_session.cc:205] T 91e0e82526c0489a8908e250bf3c851f P 1b50a031f2784ba498b3e52e16916205: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:46.200970 22487 ts_tablet_manager.cc:1875] T 91e0e82526c0489a8908e250bf3c851f P 311f765f59754bdcb4eadc04e14bd4ea: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 01:54:46.218134 22487 ts_tablet_manager.cc:1888] T 91e0e82526c0489a8908e250bf3c851f P 311f765f59754bdcb4eadc04e14bd4ea: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.8
I20240502 01:54:46.218550 22487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91e0e82526c0489a8908e250bf3c851f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:46.223444 19872 raft_consensus.cc:1261] T 62fd543d5c7c41f98926f2148ee61e7d P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 LEARNER]: Refusing update from remote peer 311f765f59754bdcb4eadc04e14bd4ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
    @     0x7fa48406ca8d  kudu::tserver::TabletCopyRunnable::Run() at ??:0
    @     0x7fa48404f981  kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
I20240502 01:54:46.236655 19738 raft_consensus.cc:1261] T 62fd543d5c7c41f98926f2148ee61e7d P 272b2179c1b4476090a89880df2a8baa [term 1 FOLLOWER]: Refusing update from remote peer 311f765f59754bdcb4eadc04e14bd4ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:46.236970 17767 raft_consensus.cc:1261] T 62fd543d5c7c41f98926f2148ee61e7d P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Refusing update from remote peer 311f765f59754bdcb4eadc04e14bd4ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:46.239677 22487 tablet_copy_client.cc:783] T 91e0e82526c0489a8908e250bf3c851f P 311f765f59754bdcb4eadc04e14bd4ea: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:46.240149 22487 tablet_copy_client.cc:647] T 91e0e82526c0489a8908e250bf3c851f P 311f765f59754bdcb4eadc04e14bd4ea: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:46.242935 22490 consensus_queue.cc:1035] T 62fd543d5c7c41f98926f2148ee61e7d P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:46.247797 22487 tablet_copy_client.cc:523] T 91e0e82526c0489a8908e250bf3c851f P 311f765f59754bdcb4eadc04e14bd4ea: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:46.250690 22197 consensus_queue.cc:1035] T 62fd543d5c7c41f98926f2148ee61e7d P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
    @     0x7fa48404f70e  std::_Function_handler<>::_M_invoke() at ??:0
    @           0xa60bcd  std::function<>::operator()() at ??:?
I20240502 01:54:46.266851 22487 tablet_bootstrap.cc:492] T 91e0e82526c0489a8908e250bf3c851f P 311f765f59754bdcb4eadc04e14bd4ea: Bootstrap starting.
    @     0x7fa474aa8b75  kudu::ThreadPool::DispatchThread() at ??:0
I20240502 01:54:46.271925 22482 consensus_queue.cc:1035] T 62fd543d5c7c41f98926f2148ee61e7d P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240502 01:54:46.292927 17432 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.176.199:42269, user_credentials={real_user=slave}} blocked reactor thread for 55184.4us
    @     0x7fa474aac19c  kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
I20240502 01:54:46.296722 22197 raft_consensus.cc:1050] T a6b5231a18a5414dbd010d4d48f08dbd P 311f765f59754bdcb4eadc04e14bd4ea: attempting to promote NON_VOTER 29806d3d37354c169484f6db368f8798 to VOTER
I20240502 01:54:46.297777 22197 consensus_queue.cc:237] T a6b5231a18a5414dbd010d4d48f08dbd P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } }
I20240502 01:54:46.298301 22490 raft_consensus.cc:2932] T 62fd543d5c7c41f98926f2148ee61e7d P 311f765f59754bdcb4eadc04e14bd4ea [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
I20240502 01:54:46.306339 19872 raft_consensus.cc:2932] T 62fd543d5c7c41f98926f2148ee61e7d P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
I20240502 01:54:46.312155 17631 raft_consensus.cc:1261] T 3b3fa060e8d64911bec810ec07a358d2 P 1b50a031f2784ba498b3e52e16916205 [term 2 FOLLOWER]: Refusing update from remote peer da3584f429104d23a1a8899b35cc2138: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240502 01:54:46.313007 22261 consensus_queue.cc:1035] T 3b3fa060e8d64911bec810ec07a358d2 P da3584f429104d23a1a8899b35cc2138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
    @     0x7fa474aac019  std::_Function_handler<>::_M_invoke() at ??:0
    @           0xa60bcd  std::function<>::operator()() at ??:?
I20240502 01:54:46.316237 22500 ts_tablet_manager.cc:902] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 272b2179c1b4476090a89880df2a8baa: Initiating tablet copy from peer 84d3a4b174c34e02aedab1e620731359 (127.9.176.195:37357)
I20240502 01:54:46.318188 17634 raft_consensus.cc:1261] T a6b5231a18a5414dbd010d4d48f08dbd P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Refusing update from remote peer 311f765f59754bdcb4eadc04e14bd4ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:46.318969 22197 consensus_queue.cc:1035] T a6b5231a18a5414dbd010d4d48f08dbd P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:46.320973 19467 raft_consensus.cc:1261] T a6b5231a18a5414dbd010d4d48f08dbd P 29806d3d37354c169484f6db368f8798 [term 1 LEARNER]: Refusing update from remote peer 311f765f59754bdcb4eadc04e14bd4ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:46.322268 19603 raft_consensus.cc:1261] T a6b5231a18a5414dbd010d4d48f08dbd P 4b7c382c9edd41babdd1a1afc904a813 [term 1 FOLLOWER]: Refusing update from remote peer 311f765f59754bdcb4eadc04e14bd4ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:46.325029 22481 raft_consensus.cc:2932] T fadcc8fe6ae047c1b142514aad8bebba P 311f765f59754bdcb4eadc04e14bd4ea [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
I20240502 01:54:46.330430 22206 consensus_queue.cc:1035] T a6b5231a18a5414dbd010d4d48f08dbd P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:46.331001 21467 consensus_queue.cc:1035] T a6b5231a18a5414dbd010d4d48f08dbd P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:46.332895 17897 raft_consensus.cc:2932] T fadcc8fe6ae047c1b142514aad8bebba P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
I20240502 01:54:46.337580 19736 raft_consensus.cc:1261] T 3b3fa060e8d64911bec810ec07a358d2 P 272b2179c1b4476090a89880df2a8baa [term 2 FOLLOWER]: Refusing update from remote peer da3584f429104d23a1a8899b35cc2138: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240502 01:54:46.338371 22261 consensus_queue.cc:1035] T 3b3fa060e8d64911bec810ec07a358d2 P da3584f429104d23a1a8899b35cc2138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 01:54:46.352707 22206 raft_consensus.cc:2932] T a6b5231a18a5414dbd010d4d48f08dbd P 311f765f59754bdcb4eadc04e14bd4ea [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 29806d3d37354c169484f6db368f8798 (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } }
I20240502 01:54:46.353561 19467 raft_consensus.cc:2932] T a6b5231a18a5414dbd010d4d48f08dbd P 29806d3d37354c169484f6db368f8798 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 29806d3d37354c169484f6db368f8798 (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } }
I20240502 01:54:46.361593 19738 raft_consensus.cc:2932] T 62fd543d5c7c41f98926f2148ee61e7d P 272b2179c1b4476090a89880df2a8baa [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
I20240502 01:54:46.362596 17350 catalog_manager.cc:5484] T a6b5231a18a5414dbd010d4d48f08dbd P 29806d3d37354c169484f6db368f8798 reported cstate change: config changed from index 7 to 8, 29806d3d37354c169484f6db368f8798 (127.9.176.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "311f765f59754bdcb4eadc04e14bd4ea" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } }
I20240502 01:54:46.364028 17352 catalog_manager.cc:5484] T fadcc8fe6ae047c1b142514aad8bebba P da3584f429104d23a1a8899b35cc2138 reported cstate change: config changed from index 4 to 5, 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "311f765f59754bdcb4eadc04e14bd4ea" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
I20240502 01:54:46.371718 21844 tablet_copy_client.cc:308] T 17365abbb6c1430aa278719eb8e4a648 P 29806d3d37354c169484f6db368f8798: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.196:46381
    @     0x7fa474a8701e  kudu::Thread::SuperviseThread() at ??:0
I20240502 01:54:46.377557 19603 raft_consensus.cc:2932] T a6b5231a18a5414dbd010d4d48f08dbd P 4b7c382c9edd41babdd1a1afc904a813 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 29806d3d37354c169484f6db368f8798 (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } }
    @     0x7fa477e0c6db  start_thread at ??:0
    @     0x7fa470c4a71f  clone at ??:0
I20240502 01:54:46.383257 17922 tablet_copy_service.cc:140] P da3584f429104d23a1a8899b35cc2138: Received BeginTabletCopySession request for tablet 17365abbb6c1430aa278719eb8e4a648 from peer 29806d3d37354c169484f6db368f8798 ({username='slave'} at 127.9.176.197:44989)
I20240502 01:54:46.383495 17922 tablet_copy_service.cc:161] P da3584f429104d23a1a8899b35cc2138: Beginning new tablet copy session on tablet 17365abbb6c1430aa278719eb8e4a648 from peer 29806d3d37354c169484f6db368f8798 at {username='slave'} at 127.9.176.197:44989: session id = 29806d3d37354c169484f6db368f8798-17365abbb6c1430aa278719eb8e4a648
I20240502 01:54:46.384325 17631 raft_consensus.cc:2932] T a6b5231a18a5414dbd010d4d48f08dbd P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 29806d3d37354c169484f6db368f8798 (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } }
I20240502 01:54:46.386673 17922 tablet_copy_source_session.cc:205] T 17365abbb6c1430aa278719eb8e4a648 P da3584f429104d23a1a8899b35cc2138: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:46.389410 17499 consensus_queue.cc:237] T a6b5231a18a5414dbd010d4d48f08dbd P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } }
I20240502 01:54:46.392113 19467 raft_consensus.cc:1261] T a6b5231a18a5414dbd010d4d48f08dbd P 29806d3d37354c169484f6db368f8798 [term 1 FOLLOWER]: Refusing update from remote peer 311f765f59754bdcb4eadc04e14bd4ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:54:46.394095 21844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17365abbb6c1430aa278719eb8e4a648. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:46.398579 17350 catalog_manager.cc:5484] T 62fd543d5c7c41f98926f2148ee61e7d P 311f765f59754bdcb4eadc04e14bd4ea reported cstate change: config changed from index 7 to 8, 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "311f765f59754bdcb4eadc04e14bd4ea" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:46.401078 19467 raft_consensus.cc:1203] T 38a60d7db47149ff90444e5276fc8935 P 29806d3d37354c169484f6db368f8798 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9]   Dedup: 1.9->[]
I20240502 01:54:46.406865 22206 consensus_queue.cc:1035] T a6b5231a18a5414dbd010d4d48f08dbd P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:46.405756 17767 raft_consensus.cc:2932] T 62fd543d5c7c41f98926f2148ee61e7d P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
I20240502 01:54:46.419981 17499 consensus_queue.cc:237] T 62fd543d5c7c41f98926f2148ee61e7d P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } }
I20240502 01:54:46.427392 22482 raft_consensus.cc:2932] T a6b5231a18a5414dbd010d4d48f08dbd P 311f765f59754bdcb4eadc04e14bd4ea [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 1b50a031f2784ba498b3e52e16916205 (127.9.176.194) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } }
I20240502 01:54:46.433002 19869 raft_consensus.cc:1261] T 62fd543d5c7c41f98926f2148ee61e7d P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Refusing update from remote peer 311f765f59754bdcb4eadc04e14bd4ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:54:46.434650 22509 consensus_queue.cc:1035] T 62fd543d5c7c41f98926f2148ee61e7d P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:46.437085 22542 ts_tablet_manager.cc:902] T b40f968655774ce887dff8b7f4651501 P 29806d3d37354c169484f6db368f8798: Initiating tablet copy from peer 1b50a031f2784ba498b3e52e16916205 (127.9.176.194:45617)
I20240502 01:54:46.441737 17764 raft_consensus.cc:2932] T fadcc8fe6ae047c1b142514aad8bebba P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
I20240502 01:54:46.441825 22509 raft_consensus.cc:2932] T 62fd543d5c7c41f98926f2148ee61e7d P 311f765f59754bdcb4eadc04e14bd4ea [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 84d3a4b174c34e02aedab1e620731359 (127.9.176.195) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
I20240502 01:54:46.453140 17343 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a6b5231a18a5414dbd010d4d48f08dbd with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:46.453254 19736 raft_consensus.cc:1261] T 62fd543d5c7c41f98926f2148ee61e7d P 272b2179c1b4476090a89880df2a8baa [term 1 FOLLOWER]: Refusing update from remote peer 311f765f59754bdcb4eadc04e14bd4ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
F20240502 01:54:46.438258 22542 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-0dd8944f9fd542e29072ca501df5e112.queue_time_us, tablet-download-pool-0dd8944f9fd542e29072ca501df5e112.queue_time_us) (tablet-download-pool-0dd8944f9fd542e29072ca501df5e112.run_cpu_time_us, tablet-download-pool-0dd8944f9fd542e29072ca501df5e112.run_cpu_time_us) (tablet-download-pool-0dd8944f9fd542e29072ca501df5e112.run_wall_time_us, tablet-download-pool-0dd8944f9fd542e29072ca501df5e112.run_wall_time_us) (tablet-download-pool-17365abbb6c1430aa278719eb8e4a648.queue_time_us, tablet-download-pool-17365abbb6c1430aa278719eb8e4a648.queue_time_us) (tablet-download-pool-17365abbb6c1430aa278719eb8e4a648.run_cpu_time_us, tablet-download-pool-17365abbb6c1430aa278719eb8e4a648.run_cpu_time_us) (tablet-download-pool-17365abbb6c1430aa278719eb8e4a648.run_wall_time_us, tablet-download-pool-17365abbb6c1430aa278719eb8e4a648.run_wall_time_us) (tablet-download-pool-2045e08a200148c69326351f38fd484d.queue_time_us, tablet-download-pool-2045e08a200148c69326351f38fd484d.queue_time_us) (tablet-download-pool-2045e08a200148c69326351f38fd484d.run_cpu_time_us, tablet-download-pool-2045e08a200148c69326351f38fd484d.run_cpu_time_us) (tablet-download-pool-2045e08a200148c69326351f38fd484d.run_wall_time_us, tablet-download-pool-2045e08a200148c69326351f38fd484d.run_wall_time_us) (tablet-download-pool-2c78951695284b1aaab411c0a041c28a.queue_time_us, tablet-download-pool-2c78951695284b1aaab411c0a041c28a.queue_time_us) (tablet-download-pool-2c78951695284b1aaab411c0a041c28a.run_cpu_time_us, tablet-download-pool-2c78951695284b1aaab411c0a041c28a.run_cpu_time_us) (tablet-download-pool-2c78951695284b1aaab411c0a041c28a.run_wall_time_us, tablet-download-pool-2c78951695284b1aaab411c0a041c28a.run_wall_time_us) (tablet-download-pool-37eda67598fe491bbb29aa62b2c4bec7.queue_time_us, tablet-download-pool-37eda67598fe491bbb29aa62b2c4bec7.queue_time_us) (tablet-download-pool-37eda67598fe491bbb29aa62b2c4bec7.run_cpu_time_us, tablet-download-pool-37eda67598fe491bbb29aa62b2c4bec7.run_cpu_time_us) (tablet-download-pool-37eda67598fe491bbb29aa62b2c4bec7.run_wall_time_us, tablet-download-pool-37eda67598fe491bbb29aa62b2c4bec7.run_wall_time_us) (tablet-download-pool-38a60d7db47149ff90444e5276fc8935.queue_time_us, tablet-download-pool-38a60d7db47149ff90444e5276fc8935.queue_time_us) (tablet-download-pool-38a60d7db47149ff90444e5276fc8935.run_cpu_time_us, tablet-download-pool-38a60d7db47149ff90444e5276fc8935.run_cpu_time_us) (tablet-download-pool-38a60d7db47149ff90444e5276fc8935.run_wall_time_us, tablet-download-pool-38a60d7db47149ff90444e5276fc8935.run_wall_time_us) (tablet-download-pool-3ca5c4988ff74e3ca8d7a5614bcb93e2.queue_time_us, tablet-download-pool-3ca5c4988ff74e3ca8d7a5614bcb93e2.queue_time_us) (tablet-download-pool-3ca5c4988ff74e3ca8d7a5614bcb93e2.run_cpu_time_us, tablet-download-pool-3ca5c4988ff74e3ca8d7a5614bcb93e2.run_cpu_time_us) (tablet-download-pool-3ca5c4988ff74e3ca8d7a5614bcb93e2.run_wall_time_us, tablet-download-pool-3ca5c4988ff74e3ca8d7a5614bcb93e2.run_wall_time_us) (tablet-download-pool-49ef898a43704b87ba077cd7b0485e9a.queue_time_us, tablet-download-pool-49ef898a43704b87ba077cd7b0485e9a.queue_time_us) (tablet-download-pool-49ef898a43704b87ba077cd7b0485e9a.run_cpu_time_us, tablet-download-pool-49ef898a43704b87ba077cd7b0485e9a.run_cpu_time_us) (tablet-download-pool-49ef898a43704b87ba077cd7b0485e9a.run_wall_time_us, tablet-download-pool-49ef898a43704b87ba077cd7b0485e9a.run_wall_time_us) (tablet-download-pool-6da59c0626c64a5988e39a831c42c268.queue_time_us, tablet-download-pool-6da59c0626c64a5988e39a831c42c268.queue_time_us) (tablet-download-pool-6da59c0626c64a5988e39a831c42c268.run_cpu_time_us, tablet-download-pool-6da59c0626c64a5988e39a831c42c268.run_cpu_time_us) (tablet-download-pool-6da59c0626c64a5988e39a831c42c268.run_wall_time_us, tablet-download-pool-6da59c0626c64a5988e39a831c42c268.run_wall_time_us) (tablet-download-pool-91e0e82526c0489a8908e250bf3c851f.queue_time_us, tablet-download-pool-91e0e82526c0489a8908e250bf3c851f.queue_time_us) (tablet-download-pool-91e0e82526c0489a8908e250bf3c851f.run_cpu_time_us, tablet-download-pool-91e0e82526c0489a8908e250bf3c851f.run_cpu_time_us) (tablet-download-pool-91e0e82526c0489a8908e250bf3c851f.run_wall_time_us, tablet-download-pool-91e0e82526c0489a8908e250bf3c851f.run_wall_time_us) (tablet-download-pool-974f67157aa04d3e8b49b264cfa104e8.queue_time_us, tablet-download-pool-974f67157aa04d3e8b49b264cfa104e8.queue_time_us) (tablet-download-pool-974f67157aa04d3e8b49b264cfa104e8.run_cpu_time_us, tablet-download-pool-974f67157aa04d3e8b49b264cfa104e8.run_cpu_time_us) (tablet-download-pool-974f67157aa04d3e8b49b264cfa104e8.run_wall_time_us, tablet-download-pool-974f67157aa04d3e8b49b264cfa104e8.run_wall_time_us) (tablet-download-pool-a5e03cf44eab41cc842342a138c04067.queue_time_us, tablet-download-pool-a5e03cf44eab41cc842342a138c04067.queue_time_us) (tablet-download-pool-a5e03cf44eab41cc842342a138c04067.run_cpu_time_us, tablet-download-pool-a5e03cf44eab41cc842342a138c04067.run_cpu_time_us) (tablet-download-pool-a5e03cf44eab41cc842342a138c04067.run_wall_time_us, tablet-download-pool-a5e03cf44eab41cc842342a138c04067.run_wall_time_us) (tablet-download-pool-a6b5231a18a5414dbd010d4d48f08dbd.queue_time_us, tablet-download-pool-a6b5231a18a5414dbd010d4d48f08dbd.queue_time_us) (tablet-download-pool-a6b5231a18a5414dbd010d4d48f08dbd.run_cpu_time_us, tablet-download-pool-a6b5231a18a5414dbd010d4d48f08dbd.run_cpu_time_us) (tablet-download-pool-a6b5231a18a5414dbd010d4d48f08dbd.run_wall_time_us, tablet-download-pool-a6b5231a18a5414dbd010d4d48f08dbd.run_wall_time_us) (tablet-download-pool-b40f968655774ce887dff8b7f4651501.queue_time_us, tablet-download-pool-b40f968655774ce887dff8b7f4651501.queue_time_us) (tablet-download-pool-c91f74f46b04425aa5393515cb9f781c.queue_time_us, tablet-download-pool-c91f74f46b04425aa5393515cb9f781c.queue_time_us) (tablet-download-pool-c91f74f46b04425aa5393515cb9f781c.run_cpu_time_us, tablet-download-pool-c91f74f46b04425aa5393515cb9f781c.run_cpu_time_us) (tablet-download-pool-c91f74f46b04425aa5393515cb9f781c.run_wall_time_us, tablet-download-pool-c91f74f46b04425aa5393515cb9f781c.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
I20240502 01:54:46.453857 17343 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 62fd543d5c7c41f98926f2148ee61e7d with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
*** Check failure stack trace: ***
I20240502 01:54:46.457896 17350 catalog_manager.cc:5484] T a6b5231a18a5414dbd010d4d48f08dbd P 311f765f59754bdcb4eadc04e14bd4ea reported cstate change: config changed from index 8 to 9, VOTER 1b50a031f2784ba498b3e52e16916205 (127.9.176.194) evicted. New cstate: current_term: 1 leader_uuid: "311f765f59754bdcb4eadc04e14bd4ea" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:46.460153 17897 consensus_queue.cc:237] T 3b3fa060e8d64911bec810ec07a358d2 P da3584f429104d23a1a8899b35cc2138 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 11, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } }
I20240502 01:54:46.463024 22500 tablet_copy_client.cc:308] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 272b2179c1b4476090a89880df2a8baa: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.195:37357
I20240502 01:54:46.464354 17789 tablet_copy_service.cc:140] P 84d3a4b174c34e02aedab1e620731359: Received BeginTabletCopySession request for tablet 3ca5c4988ff74e3ca8d7a5614bcb93e2 from peer 272b2179c1b4476090a89880df2a8baa ({username='slave'} at 127.9.176.199:46159)
I20240502 01:54:46.464591 17789 tablet_copy_service.cc:161] P 84d3a4b174c34e02aedab1e620731359: Beginning new tablet copy session on tablet 3ca5c4988ff74e3ca8d7a5614bcb93e2 from peer 272b2179c1b4476090a89880df2a8baa at {username='slave'} at 127.9.176.199:46159: session id = 272b2179c1b4476090a89880df2a8baa-3ca5c4988ff74e3ca8d7a5614bcb93e2
I20240502 01:54:46.467229 17499 consensus_queue.cc:237] T fadcc8fe6ae047c1b142514aad8bebba P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } }
I20240502 01:54:46.468868 17789 tablet_copy_source_session.cc:205] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 84d3a4b174c34e02aedab1e620731359: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:46.470258 22482 consensus_queue.cc:1035] T 62fd543d5c7c41f98926f2148ee61e7d P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:46.471478 19467 raft_consensus.cc:2932] T a6b5231a18a5414dbd010d4d48f08dbd P 29806d3d37354c169484f6db368f8798 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 1b50a031f2784ba498b3e52e16916205 (127.9.176.194) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } }
I20240502 01:54:46.476343 17897 raft_consensus.cc:1261] T fadcc8fe6ae047c1b142514aad8bebba P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Refusing update from remote peer 311f765f59754bdcb4eadc04e14bd4ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:46.476461 19736 raft_consensus.cc:1261] T 3b3fa060e8d64911bec810ec07a358d2 P 272b2179c1b4476090a89880df2a8baa [term 2 FOLLOWER]: Refusing update from remote peer da3584f429104d23a1a8899b35cc2138: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240502 01:54:46.477442 22546 consensus_queue.cc:1035] T 3b3fa060e8d64911bec810ec07a358d2 P da3584f429104d23a1a8899b35cc2138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 01:54:46.477156 19872 raft_consensus.cc:2932] T 62fd543d5c7c41f98926f2148ee61e7d P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 84d3a4b174c34e02aedab1e620731359 (127.9.176.195) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
I20240502 01:54:46.480419 22500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ca5c4988ff74e3ca8d7a5614bcb93e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:46.481638 19737 raft_consensus.cc:2932] T 62fd543d5c7c41f98926f2148ee61e7d P 272b2179c1b4476090a89880df2a8baa [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 84d3a4b174c34e02aedab1e620731359 (127.9.176.195) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
I20240502 01:54:46.489899 22534 raft_consensus.cc:2932] T 3b3fa060e8d64911bec810ec07a358d2 P da3584f429104d23a1a8899b35cc2138 [term 2 LEADER]: Committing config change with OpId 2.13: config changed from index 11 to 13, VOTER 1b50a031f2784ba498b3e52e16916205 (127.9.176.194) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
I20240502 01:54:46.497313 19737 raft_consensus.cc:2932] T 3b3fa060e8d64911bec810ec07a358d2 P 272b2179c1b4476090a89880df2a8baa [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 11 to 13, VOTER 1b50a031f2784ba498b3e52e16916205 (127.9.176.194) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
I20240502 01:54:46.498618 22481 consensus_queue.cc:1035] T fadcc8fe6ae047c1b142514aad8bebba P 311f765f59754bdcb4eadc04e14bd4ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.017s
I20240502 01:54:46.504560 22482 raft_consensus.cc:2932] T fadcc8fe6ae047c1b142514aad8bebba P 311f765f59754bdcb4eadc04e14bd4ea [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 84d3a4b174c34e02aedab1e620731359 (127.9.176.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
    @     0x7fea44324dcd  google::LogMessage::Fail() at ??:0
I20240502 01:54:46.509164 22441 tablet_bootstrap.cc:492] T a5e03cf44eab41cc842342a138c04067 P 272b2179c1b4476090a89880df2a8baa: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
    @     0x7fea44328b93  google::LogMessage::SendToLog() at ??:0
I20240502 01:54:46.510900 17897 raft_consensus.cc:2932] T fadcc8fe6ae047c1b142514aad8bebba P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 84d3a4b174c34e02aedab1e620731359 (127.9.176.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
I20240502 01:54:46.513772 21844 tablet_copy_client.cc:783] T 17365abbb6c1430aa278719eb8e4a648 P 29806d3d37354c169484f6db368f8798: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:46.514081 21844 tablet_copy_client.cc:647] T 17365abbb6c1430aa278719eb8e4a648 P 29806d3d37354c169484f6db368f8798: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:46.519039 17354 catalog_manager.cc:5484] T 3b3fa060e8d64911bec810ec07a358d2 P da3584f429104d23a1a8899b35cc2138 reported cstate change: config changed from index 11 to 13, VOTER 1b50a031f2784ba498b3e52e16916205 (127.9.176.194) evicted. New cstate: current_term: 2 leader_uuid: "da3584f429104d23a1a8899b35cc2138" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:46.519311 17347 catalog_manager.cc:5484] T 62fd543d5c7c41f98926f2148ee61e7d P 94931f4b51c9493cb3d4264ce99f5e76 reported cstate change: config changed from index 8 to 9, VOTER 84d3a4b174c34e02aedab1e620731359 (127.9.176.195) evicted. New cstate: current_term: 1 leader_uuid: "311f765f59754bdcb4eadc04e14bd4ea" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } }
    @     0x7fea443247cc  google::LogMessage::Flush() at ??:0
I20240502 01:54:46.529541 22441 tablet_bootstrap.cc:492] T a5e03cf44eab41cc842342a138c04067 P 272b2179c1b4476090a89880df2a8baa: Bootstrap complete.
I20240502 01:54:46.531090 22441 ts_tablet_manager.cc:1367] T a5e03cf44eab41cc842342a138c04067 P 272b2179c1b4476090a89880df2a8baa: Time spent bootstrapping tablet: real 0.392s	user 0.059s	sys 0.006s
I20240502 01:54:46.530123 17340 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3b3fa060e8d64911bec810ec07a358d2 with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:46.532589 17343 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet fadcc8fe6ae047c1b142514aad8bebba with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:46.534615 21844 tablet_copy_client.cc:523] T 17365abbb6c1430aa278719eb8e4a648 P 29806d3d37354c169484f6db368f8798: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:46.536242 22441 raft_consensus.cc:348] T a5e03cf44eab41cc842342a138c04067 P 272b2179c1b4476090a89880df2a8baa [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } }
I20240502 01:54:46.536554 22441 raft_consensus.cc:729] T a5e03cf44eab41cc842342a138c04067 P 272b2179c1b4476090a89880df2a8baa [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 272b2179c1b4476090a89880df2a8baa, State: Initialized, Role: LEARNER
I20240502 01:54:46.536774 22441 consensus_queue.cc:260] T a5e03cf44eab41cc842342a138c04067 P 272b2179c1b4476090a89880df2a8baa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } }
I20240502 01:54:46.537842 17746 tablet_service.cc:1514] Processing DeleteTablet for tablet 62fd543d5c7c41f98926f2148ee61e7d with delete_type TABLET_DATA_TOMBSTONED (TS 84d3a4b174c34e02aedab1e620731359 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:47166
I20240502 01:54:46.539551 17613 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b3fa060e8d64911bec810ec07a358d2 with delete_type TABLET_DATA_TOMBSTONED (TS 1b50a031f2784ba498b3e52e16916205 not found in new config with opid_index 13) from {username='slave'} at 127.0.0.1:41484
I20240502 01:54:46.540591 17347 catalog_manager.cc:5484] T fadcc8fe6ae047c1b142514aad8bebba P da3584f429104d23a1a8899b35cc2138 reported cstate change: config changed from index 5 to 6, VOTER 84d3a4b174c34e02aedab1e620731359 (127.9.176.195) evicted. New cstate: current_term: 1 leader_uuid: "311f765f59754bdcb4eadc04e14bd4ea" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } } }
I20240502 01:54:46.552659 17745 tablet_service.cc:1514] Processing DeleteTablet for tablet fadcc8fe6ae047c1b142514aad8bebba with delete_type TABLET_DATA_TOMBSTONED (TS 84d3a4b174c34e02aedab1e620731359 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47166
I20240502 01:54:46.562054 17612 tablet_service.cc:1514] Processing DeleteTablet for tablet a6b5231a18a5414dbd010d4d48f08dbd with delete_type TABLET_DATA_TOMBSTONED (TS 1b50a031f2784ba498b3e52e16916205 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:41484
I20240502 01:54:46.563771 21844 tablet_bootstrap.cc:492] T 17365abbb6c1430aa278719eb8e4a648 P 29806d3d37354c169484f6db368f8798: Bootstrap starting.
    @     0x7fea44325f59  google::LogMessageFatal::~LogMessageFatal() at ??:0
I20240502 01:54:46.579826 22441 ts_tablet_manager.cc:1398] T a5e03cf44eab41cc842342a138c04067 P 272b2179c1b4476090a89880df2a8baa: Time spent starting tablet: real 0.048s	user 0.002s	sys 0.000s
I20240502 01:54:46.580015 22500 tablet_copy_client.cc:783] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 272b2179c1b4476090a89880df2a8baa: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:46.580338 22500 tablet_copy_client.cc:647] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 272b2179c1b4476090a89880df2a8baa: tablet copy: Starting download of 1 WAL segments...
    @     0x7fea47065742  kudu::TraceMetrics::InternName() at ??:0
I20240502 01:54:46.581660 22555 tablet_replica.cc:330] T 3b3fa060e8d64911bec810ec07a358d2 P 1b50a031f2784ba498b3e52e16916205: stopping tablet replica
I20240502 01:54:46.581728 17922 tablet_copy_service.cc:342] P da3584f429104d23a1a8899b35cc2138: Request end of tablet copy session 272b2179c1b4476090a89880df2a8baa-a5e03cf44eab41cc842342a138c04067 received from {username='slave'} at 127.9.176.199:46107
I20240502 01:54:46.581856 17922 tablet_copy_service.cc:433] P da3584f429104d23a1a8899b35cc2138: ending tablet copy session 272b2179c1b4476090a89880df2a8baa-a5e03cf44eab41cc842342a138c04067 on tablet a5e03cf44eab41cc842342a138c04067 with peer 272b2179c1b4476090a89880df2a8baa
I20240502 01:54:46.581909 22555 raft_consensus.cc:2229] T 3b3fa060e8d64911bec810ec07a358d2 P 1b50a031f2784ba498b3e52e16916205 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:46.582178 22555 raft_consensus.cc:2258] T 3b3fa060e8d64911bec810ec07a358d2 P 1b50a031f2784ba498b3e52e16916205 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:46.582893 22554 tablet_replica.cc:330] T 62fd543d5c7c41f98926f2148ee61e7d P 84d3a4b174c34e02aedab1e620731359: stopping tablet replica
I20240502 01:54:46.583169 22554 raft_consensus.cc:2229] T 62fd543d5c7c41f98926f2148ee61e7d P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:46.583446 22554 raft_consensus.cc:2258] T 62fd543d5c7c41f98926f2148ee61e7d P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:46.585906 22500 tablet_copy_client.cc:523] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 272b2179c1b4476090a89880df2a8baa: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:46.592196 22500 tablet_bootstrap.cc:492] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 272b2179c1b4476090a89880df2a8baa: Bootstrap starting.
    @     0x7fea47031f8b  kudu::ThreadPool::ThreadPool() at ??:0
    @     0x7fea4702bcd6  kudu::ThreadPoolBuilder::Build() at ??:0
    @     0x7fea5647e8de  kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
I20240502 01:54:46.584667 22555 ts_tablet_manager.cc:1875] T 3b3fa060e8d64911bec810ec07a358d2 P 1b50a031f2784ba498b3e52e16916205: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:46.642977 22554 ts_tablet_manager.cc:1875] T 62fd543d5c7c41f98926f2148ee61e7d P 84d3a4b174c34e02aedab1e620731359: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
    @     0x7fea56496daf  kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
I20240502 01:54:46.646425 22487 tablet_bootstrap.cc:492] T 91e0e82526c0489a8908e250bf3c851f P 311f765f59754bdcb4eadc04e14bd4ea: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:46.647008 22487 tablet_bootstrap.cc:492] T 91e0e82526c0489a8908e250bf3c851f P 311f765f59754bdcb4eadc04e14bd4ea: Bootstrap complete.
I20240502 01:54:46.647392 22487 ts_tablet_manager.cc:1367] T 91e0e82526c0489a8908e250bf3c851f P 311f765f59754bdcb4eadc04e14bd4ea: Time spent bootstrapping tablet: real 0.381s	user 0.057s	sys 0.009s
I20240502 01:54:46.649423 17631 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2c78951695284b1aaab411c0a041c28a"
dest_uuid: "1b50a031f2784ba498b3e52e16916205"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:55628
I20240502 01:54:46.649694 17631 raft_consensus.cc:595] T 2c78951695284b1aaab411c0a041c28a P 1b50a031f2784ba498b3e52e16916205 [term 1 LEADER]: Received request to transfer leadership
    @     0x7fea565c1480  kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
I20240502 01:54:46.667935 22487 raft_consensus.cc:348] T 91e0e82526c0489a8908e250bf3c851f P 311f765f59754bdcb4eadc04e14bd4ea [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: NON_VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { promote: true } }
I20240502 01:54:46.676934 22487 raft_consensus.cc:729] T 91e0e82526c0489a8908e250bf3c851f P 311f765f59754bdcb4eadc04e14bd4ea [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 311f765f59754bdcb4eadc04e14bd4ea, State: Initialized, Role: LEARNER
I20240502 01:54:46.677273 22487 consensus_queue.cc:260] T 91e0e82526c0489a8908e250bf3c851f P 311f765f59754bdcb4eadc04e14bd4ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: NON_VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { promote: true } }
I20240502 01:54:46.678054 22487 ts_tablet_manager.cc:1398] T 91e0e82526c0489a8908e250bf3c851f P 311f765f59754bdcb4eadc04e14bd4ea: Time spent starting tablet: real 0.031s	user 0.004s	sys 0.001s
I20240502 01:54:46.681949 17654 tablet_copy_service.cc:342] P 1b50a031f2784ba498b3e52e16916205: Request end of tablet copy session 311f765f59754bdcb4eadc04e14bd4ea-91e0e82526c0489a8908e250bf3c851f received from {username='slave'} at 127.9.176.193:52149
I20240502 01:54:46.682155 17654 tablet_copy_service.cc:433] P 1b50a031f2784ba498b3e52e16916205: ending tablet copy session 311f765f59754bdcb4eadc04e14bd4ea-91e0e82526c0489a8908e250bf3c851f on tablet 91e0e82526c0489a8908e250bf3c851f with peer 311f765f59754bdcb4eadc04e14bd4ea
I20240502 01:54:46.683379 17499 raft_consensus.cc:1203] T 91e0e82526c0489a8908e250bf3c851f P 311f765f59754bdcb4eadc04e14bd4ea [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10]   Dedup: 1.10->[]
    @     0x7fea565f9a8d  kudu::tserver::TabletCopyRunnable::Run() at ??:0
    @     0x7fea565dc981  kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
W20240502 01:54:46.909119 22500 log.cc:924] Time spent T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 272b2179c1b4476090a89880df2a8baa: Append to log took a long time: real 0.080s	user 0.000s	sys 0.000s
I20240502 01:54:47.002862 19718 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
    @     0x7fea565dc70e  std::_Function_handler<>::_M_invoke() at ??:0
I20240502 01:54:47.032845 17478 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
    @           0xa60bcd  std::function<>::operator()() at ??:?
I20240502 01:54:47.039971 19737 raft_consensus.cc:1203] T a5e03cf44eab41cc842342a138c04067 P 272b2179c1b4476090a89880df2a8baa [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240502 01:54:47.048107 19446 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:54:47.051196 17832 connection.cc:535] client connection to 127.9.176.198:36865 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:54:47.051501 17832 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:54:47.051527 17432 connection.cc:535] client connection to 127.9.176.198:36865 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:54:47.051908 17432 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
    @     0x7fea47035b75  kudu::ThreadPool::DispatchThread() at ??:0
W20240502 01:54:47.094292 17832 consensus_peers.cc:485] T 17365abbb6c1430aa278719eb8e4a648 P da3584f429104d23a1a8899b35cc2138 -> Peer 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198:36865): Couldn't send request to peer 4b7c382c9edd41babdd1a1afc904a813. Status: Network error: Client connection negotiation failed: client connection to 127.9.176.198:36865: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
    @     0x7fea4703919c  kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
W20240502 01:54:47.125639 17832 consensus_peers.cc:485] T 05b67d9912f047bfb6a2bed90b82ae15 P da3584f429104d23a1a8899b35cc2138 -> Peer 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198:36865): Couldn't send request to peer 4b7c382c9edd41babdd1a1afc904a813. Status: Network error: Client connection negotiation failed: client connection to 127.9.176.198:36865: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:47.127749 17832 consensus_peers.cc:485] T 3b3fa060e8d64911bec810ec07a358d2 P da3584f429104d23a1a8899b35cc2138 -> Peer 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198:36865): Couldn't send request to peer 4b7c382c9edd41babdd1a1afc904a813. Status: Network error: Client connection negotiation failed: client connection to 127.9.176.198:36865: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:47.158267 17880 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:54:47.165227 19852 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
    @     0x7fea47039019  std::_Function_handler<>::_M_invoke() at ??:0
    @           0xa60bcd  std::function<>::operator()() at ??:?
    @     0x7fea4701401e  kudu::Thread::SuperviseThread() at ??:0
    @     0x7fea4a3996db  start_thread at ??:0
    @     0x7fea431d771f  clone at ??:0
I20240502 01:54:46.771368 21926 raft_consensus.cc:979] T 2c78951695284b1aaab411c0a041c28a P 1b50a031f2784ba498b3e52e16916205: : Instructing follower 29806d3d37354c169484f6db368f8798 to start an election
I20240502 01:54:47.208971 21926 raft_consensus.cc:1067] T 2c78951695284b1aaab411c0a041c28a P 1b50a031f2784ba498b3e52e16916205 [term 1 LEADER]: Signalling peer 29806d3d37354c169484f6db368f8798 to start an election
I20240502 01:54:46.788012 22555 ts_tablet_manager.cc:1888] T 3b3fa060e8d64911bec810ec07a358d2 P 1b50a031f2784ba498b3e52e16916205: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.12
I20240502 01:54:46.830111 22290 raft_consensus.cc:1050] T 38a60d7db47149ff90444e5276fc8935 P 1b50a031f2784ba498b3e52e16916205: attempting to promote NON_VOTER 29806d3d37354c169484f6db368f8798 to VOTER
I20240502 01:54:46.742669 22554 ts_tablet_manager.cc:1888] T 62fd543d5c7c41f98926f2148ee61e7d P 84d3a4b174c34e02aedab1e620731359: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
W20240502 01:54:47.050958 17700 connection.cc:535] client connection to 127.9.176.198:36865 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 01:54:47.209431 22554 log.cc:1196] T 62fd543d5c7c41f98926f2148ee61e7d P 84d3a4b174c34e02aedab1e620731359: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/wal/wals/62fd543d5c7c41f98926f2148ee61e7d
W20240502 01:54:47.209614 17700 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 01:54:47.164461 17746 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:54:47.210446 22554 tablet_replica.cc:330] T fadcc8fe6ae047c1b142514aad8bebba P 84d3a4b174c34e02aedab1e620731359: stopping tablet replica
I20240502 01:54:47.210229 22290 consensus_queue.cc:237] T 38a60d7db47149ff90444e5276fc8935 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } }
I20240502 01:54:47.210762 22554 raft_consensus.cc:2229] T fadcc8fe6ae047c1b142514aad8bebba P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:47.211133 22554 raft_consensus.cc:2258] T fadcc8fe6ae047c1b142514aad8bebba P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240502 01:54:46.905691 17568 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.176.195:37357, user_credentials={real_user=slave}} blocked reactor thread for 48485.4us
I20240502 01:54:47.009927 17613 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:54:47.051556 17565 connection.cc:535] client connection to 127.9.176.198:36865 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:54:47.212381 17565 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 01:54:47.209224 22555 log.cc:1196] T 3b3fa060e8d64911bec810ec07a358d2 P 1b50a031f2784ba498b3e52e16916205: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/wal/wals/3b3fa060e8d64911bec810ec07a358d2
I20240502 01:54:47.214406 22555 tablet_replica.cc:330] T a6b5231a18a5414dbd010d4d48f08dbd P 1b50a031f2784ba498b3e52e16916205: stopping tablet replica
I20240502 01:54:47.214700 22555 raft_consensus.cc:2229] T a6b5231a18a5414dbd010d4d48f08dbd P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Raft consensus shutting down.
W20240502 01:54:47.235337 17432 consensus_peers.cc:485] T a6b5231a18a5414dbd010d4d48f08dbd P 311f765f59754bdcb4eadc04e14bd4ea -> Peer 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198:36865): Couldn't send request to peer 4b7c382c9edd41babdd1a1afc904a813. Status: Network error: Client connection negotiation failed: client connection to 127.9.176.198:36865: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:47.620898 17432 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.176.198:36865, user_credentials={real_user=slave}} blocked reactor thread for 385491us
W20240502 01:54:47.621209 17432 consensus_peers.cc:485] T 7ce9d4281c1b46258493feb39e8d9ab2 P 311f765f59754bdcb4eadc04e14bd4ea -> Peer 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198:36865): Couldn't send request to peer 4b7c382c9edd41babdd1a1afc904a813. Status: Network error: Client connection negotiation failed: client connection to 127.9.176.198:36865: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:47.621399 17432 consensus_peers.cc:485] T 59e507855e5043d38cef84e92adf437b P 311f765f59754bdcb4eadc04e14bd4ea -> Peer 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198:36865): Couldn't send request to peer 4b7c382c9edd41babdd1a1afc904a813. Status: Network error: Client connection negotiation failed: client connection to 127.9.176.198:36865: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:47.621487 17432 consensus_peers.cc:485] T fadcc8fe6ae047c1b142514aad8bebba P 311f765f59754bdcb4eadc04e14bd4ea -> Peer 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198:36865): Couldn't send request to peer 4b7c382c9edd41babdd1a1afc904a813. Status: Network error: Client connection negotiation failed: client connection to 127.9.176.198:36865: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:47.393388 22500 log.cc:924] Time spent T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 272b2179c1b4476090a89880df2a8baa: Append to log took a long time: real 0.155s	user 0.002s	sys 0.000s
I20240502 01:54:47.219303 17343 catalog_manager.cc:4830] TS 84d3a4b174c34e02aedab1e620731359 (127.9.176.195:37357): tablet 62fd543d5c7c41f98926f2148ee61e7d (table rebalance_test_table_0 [id=c516e7439e734f02a0947c3680cf79e2]) successfully deleted
W20240502 01:54:47.622082 17343 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.9.176.195:37357, user_credentials={real_user=slave}} blocked reactor thread for 402681us
I20240502 01:54:47.622936 22500 tablet_bootstrap.cc:492] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 272b2179c1b4476090a89880df2a8baa: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:47.237898 17499 raft_consensus.cc:1261] T 38a60d7db47149ff90444e5276fc8935 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:54:47.623586 22500 tablet_bootstrap.cc:492] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 272b2179c1b4476090a89880df2a8baa: Bootstrap complete.
W20240502 01:54:47.526974 17329 kernel_stack_watchdog.cc:198] Thread 17343 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:308 for 132ms:
Kernel stack:
(could not read kernel stack)
User stack:
    @     0x7f356dcea980  (unknown) at ??:0
    @     0x7f3566b0c255  __write at ??:0
    @     0x7f3566a8728d  _IO_file_write at ??:0
    @     0x7f3566a87bff  _IO_file_xsputn at ??:0
    @     0x7f3566a7ba57  fwrite at ??:0
    @           0x8793e7  __interceptor_fwrite.part.56 at ??:?
    @     0x7f3567c6e8a0  google::LogMessage::SendToLog() at ??:0
    @     0x7f3567c6a7cc  google::LogMessage::Flush() at ??:0
    @     0x7f3567c6aa11  google::LogMessage::~LogMessage() at ??:0
    @     0x7f357de0a1e8  kudu::master::AsyncDeleteReplica::HandleResponse() at ??:0
    @     0x7f357dd62b09  kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
    @     0x7f357de0de30  _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
    @     0x7f357de0dc4e  _ZNSt17_Function_handlerIFvvEZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
    @           0xa60bcd  std::function<>::operator()() at ??:?
    @     0x7f35715aa8cf  kudu::rpc::Proxy::AsyncRequest()::$_1::operator()() at ??:0
    @     0x7f35715aa4ae  std::_Function_handler<>::_M_invoke() at ??:0
I20240502 01:54:47.623889 22500 ts_tablet_manager.cc:1367] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 272b2179c1b4476090a89880df2a8baa: Time spent bootstrapping tablet: real 1.032s	user 0.070s	sys 0.012s
W20240502 01:54:47.493574 17420 kernel_stack_watchdog.cc:198] Thread 17432 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:308 for 135ms:
Kernel stack:
(could not read kernel stack)
User stack:
    @     0x7fedd39d3980  (unknown) at ??:0
    @     0x7fedcc7f5255  __write at ??:0
    @     0x7fedcc77028d  _IO_file_write at ??:0
    @     0x7fedcc770bff  _IO_file_xsputn at ??:0
    @     0x7fedcc764a57  fwrite at ??:0
    @           0x8793e7  __interceptor_fwrite.part.56 at ??:?
    @     0x7fedcd9578a0  google::LogMessage::SendToLog() at ??:0
    @     0x7fedcd9537cc  google::LogMessage::Flush() at ??:0
    @     0x7fedcd953a11  google::LogMessage::~LogMessage() at ??:0
    @     0x7feddbd909ee  kudu::consensus::Peer::ProcessResponseErrorUnlocked() at ??:0
    @     0x7feddbd8f69f  kudu::consensus::Peer::ProcessResponse() at ??:0
    @     0x7feddbd96061  kudu::consensus::Peer::SendNextRequest()::$_3::operator()() at ??:0
    @     0x7feddbd95e6e  std::_Function_handler<>::_M_invoke() at ??:0
    @           0xa60bcd  std::function<>::operator()() at ??:?
    @     0x7fedd727f4e0  kudu::rpc::OutboundCall::CallCallback() at ??:0
    @     0x7fedd7280365  kudu::rpc::OutboundCall::SetFailed() at ??:0
I20240502 01:54:47.624861 22500 raft_consensus.cc:348] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 272b2179c1b4476090a89880df2a8baa [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } }
I20240502 01:54:47.625229 22500 raft_consensus.cc:729] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 272b2179c1b4476090a89880df2a8baa [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 272b2179c1b4476090a89880df2a8baa, State: Initialized, Role: LEARNER
I20240502 01:54:47.625504 22500 consensus_queue.cc:260] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 272b2179c1b4476090a89880df2a8baa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } }
I20240502 01:54:47.229694 19872 raft_consensus.cc:1261] T 38a60d7db47149ff90444e5276fc8935 P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:54:47.225020 22554 ts_tablet_manager.cc:1875] T fadcc8fe6ae047c1b142514aad8bebba P 84d3a4b174c34e02aedab1e620731359: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:47.630400 22546 raft_consensus.cc:1050] T a5e03cf44eab41cc842342a138c04067 P da3584f429104d23a1a8899b35cc2138: attempting to promote NON_VOTER 272b2179c1b4476090a89880df2a8baa to VOTER
I20240502 01:54:47.215067 22555 raft_consensus.cc:2258] T a6b5231a18a5414dbd010d4d48f08dbd P 1b50a031f2784ba498b3e52e16916205 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:47.631482 22555 ts_tablet_manager.cc:1875] T a6b5231a18a5414dbd010d4d48f08dbd P 1b50a031f2784ba498b3e52e16916205: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:47.631323 22546 consensus_queue.cc:237] T a5e03cf44eab41cc842342a138c04067 P da3584f429104d23a1a8899b35cc2138 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } }
I20240502 01:54:47.222002 22600 raft_consensus.cc:1050] T 91e0e82526c0489a8908e250bf3c851f P 1b50a031f2784ba498b3e52e16916205: attempting to promote NON_VOTER 311f765f59754bdcb4eadc04e14bd4ea to VOTER
I20240502 01:54:47.633590 22600 consensus_queue.cc:237] T 91e0e82526c0489a8908e250bf3c851f P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { promote: false } }
W20240502 01:54:47.257681 17700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.176.198:36865, user_credentials={real_user=slave}} blocked reactor thread for 48188.1us
W20240502 01:54:47.232878 17565 consensus_peers.cc:462] T df16e178e04040699201f325ce9e2979 P 1b50a031f2784ba498b3e52e16916205 -> Peer 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198:36865): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.9.176.198:36865: connect: Connection refused (error 111)
W20240502 01:54:47.635610 17565 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.StartTabletCopy -> {remote=127.9.176.198:36865, user_credentials={real_user=slave}} blocked reactor thread for 402692us
W20240502 01:54:47.635908 17565 consensus_peers.cc:485] T 5f3f3e5f772b4566b69236fd87d71a38 P 1b50a031f2784ba498b3e52e16916205 -> Peer 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198:36865): Couldn't send request to peer 4b7c382c9edd41babdd1a1afc904a813. Status: Network error: Client connection negotiation failed: client connection to 127.9.176.198:36865: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:47.618068 17613 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:54:47.638630 22500 ts_tablet_manager.cc:1398] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 272b2179c1b4476090a89880df2a8baa: Time spent starting tablet: real 0.015s	user 0.000s	sys 0.002s
I20240502 01:54:47.645114 17789 tablet_copy_service.cc:342] P 84d3a4b174c34e02aedab1e620731359: Request end of tablet copy session 272b2179c1b4476090a89880df2a8baa-3ca5c4988ff74e3ca8d7a5614bcb93e2 received from {username='slave'} at 127.9.176.199:46159
I20240502 01:54:47.645301 17789 tablet_copy_service.cc:433] P 84d3a4b174c34e02aedab1e620731359: ending tablet copy session 272b2179c1b4476090a89880df2a8baa-3ca5c4988ff74e3ca8d7a5614bcb93e2 on tablet 3ca5c4988ff74e3ca8d7a5614bcb93e2 with peer 272b2179c1b4476090a89880df2a8baa
I20240502 01:54:47.646711 22554 ts_tablet_manager.cc:1888] T fadcc8fe6ae047c1b142514aad8bebba P 84d3a4b174c34e02aedab1e620731359: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:54:47.646854 22554 log.cc:1196] T fadcc8fe6ae047c1b142514aad8bebba P 84d3a4b174c34e02aedab1e620731359: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/wal/wals/fadcc8fe6ae047c1b142514aad8bebba
I20240502 01:54:47.647439 22554 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47166 (request call id 46) took 1094 ms. Trace:
I20240502 01:54:47.627009 21926 consensus_queue.cc:1035] T 38a60d7db47149ff90444e5276fc8935 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:54:47.647624 22554 rpcz_store.cc:276] 0502 01:54:46.552482 (+     0us) service_pool.cc:168] Inserting onto call queue
0502 01:54:46.552548 (+    66us) service_pool.cc:225] Handling call
0502 01:54:47.210235 (+657687us) ts_tablet_manager.cc:1120] Deleting tablet fadcc8fe6ae047c1b142514aad8bebba
0502 01:54:47.210248 (+    13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 01:54:47.629880 (+419632us) tablet_metadata.cc:641] Metadata flushed
0502 01:54:47.646547 (+ 16667us) tablet_metadata.cc:641] Metadata flushed
0502 01:54:47.647367 (+   820us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":657360}
I20240502 01:54:47.650872 22555 ts_tablet_manager.cc:1888] T a6b5231a18a5414dbd010d4d48f08dbd P 1b50a031f2784ba498b3e52e16916205: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 01:54:47.651039 22555 log.cc:1196] T a6b5231a18a5414dbd010d4d48f08dbd P 1b50a031f2784ba498b3e52e16916205: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/wal/wals/a6b5231a18a5414dbd010d4d48f08dbd
I20240502 01:54:47.647910 22603 consensus_queue.cc:1035] T 38a60d7db47149ff90444e5276fc8935 P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.002s
I20240502 01:54:47.653355 22555 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41484 (request call id 48) took 1091 ms. Trace:
I20240502 01:54:47.653502 22555 rpcz_store.cc:276] 0502 01:54:46.561766 (+     0us) service_pool.cc:168] Inserting onto call queue
0502 01:54:46.561878 (+   112us) service_pool.cc:225] Handling call
0502 01:54:47.214163 (+652285us) ts_tablet_manager.cc:1120] Deleting tablet a6b5231a18a5414dbd010d4d48f08dbd
0502 01:54:47.214173 (+    10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 01:54:47.637499 (+423326us) tablet_metadata.cc:641] Metadata flushed
0502 01:54:47.650676 (+ 13177us) tablet_metadata.cc:641] Metadata flushed
0502 01:54:47.653281 (+  2605us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":651850}
I20240502 01:54:47.657586 17343 catalog_manager.cc:4830] TS 1b50a031f2784ba498b3e52e16916205 (127.9.176.194:45617): tablet 3b3fa060e8d64911bec810ec07a358d2 (table rebalance_test_table_2 [id=bf67e64a43cd4367a7420c6bf1793faa]) successfully deleted
I20240502 01:54:47.657927 17343 catalog_manager.cc:4830] TS 1b50a031f2784ba498b3e52e16916205 (127.9.176.194:45617): tablet a6b5231a18a5414dbd010d4d48f08dbd (table rebalance_test_table_6 [id=d49bf19a92cf45ceadf02d01dbdae6ad]) successfully deleted
I20240502 01:54:47.669624 19737 raft_consensus.cc:1261] T a5e03cf44eab41cc842342a138c04067 P 272b2179c1b4476090a89880df2a8baa [term 1 LEARNER]: Refusing update from remote peer da3584f429104d23a1a8899b35cc2138: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:47.670372 22539 consensus_queue.cc:1035] T a5e03cf44eab41cc842342a138c04067 P da3584f429104d23a1a8899b35cc2138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:47.679252 17767 raft_consensus.cc:1261] T a5e03cf44eab41cc842342a138c04067 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Refusing update from remote peer da3584f429104d23a1a8899b35cc2138: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:47.687922 22546 consensus_queue.cc:1035] T a5e03cf44eab41cc842342a138c04067 P da3584f429104d23a1a8899b35cc2138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:47.719509 19718 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:47.744546 17343 catalog_manager.cc:4830] TS 84d3a4b174c34e02aedab1e620731359 (127.9.176.195:37357): tablet fadcc8fe6ae047c1b142514aad8bebba (table rebalance_test_table_8 [id=40a6be2c1a2744428cea56b78225e44e]) successfully deleted
W20240502 01:54:47.750217 17565 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.176.199:42269, user_credentials={real_user=slave}} blocked reactor thread for 113397us
W20240502 01:54:47.752213 17700 consensus_peers.cc:485] T f07e9b433e554c85b3c25b7e708f8b2f P 84d3a4b174c34e02aedab1e620731359 -> Peer 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198:36865): Couldn't send request to peer 4b7c382c9edd41babdd1a1afc904a813. Status: Network error: Client connection negotiation failed: client connection to 127.9.176.198:36865: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:47.764770 19736 raft_consensus.cc:1203] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 272b2179c1b4476090a89880df2a8baa [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
W20240502 01:54:47.766870 17432 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.176.198:36865, user_credentials={real_user=slave}} blocked reactor thread for 71531.9us
W20240502 01:54:47.752924 17700 consensus_peers.cc:485] T 88a137bcfc76410583f910b377d1be7f P 84d3a4b174c34e02aedab1e620731359 -> Peer 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198:36865): Couldn't send request to peer 4b7c382c9edd41babdd1a1afc904a813. Status: Network error: Client connection negotiation failed: client connection to 127.9.176.198:36865: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:47.780596 17700 consensus_peers.cc:485] T c91f74f46b04425aa5393515cb9f781c P 84d3a4b174c34e02aedab1e620731359 -> Peer 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198:36865): Couldn't send request to peer 4b7c382c9edd41babdd1a1afc904a813. Status: Network error: Client connection negotiation failed: client connection to 127.9.176.198:36865: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:47.780748 17700 consensus_peers.cc:485] T 768e7cbf63a94d3a815ac77a4d0acf41 P 84d3a4b174c34e02aedab1e620731359 -> Peer 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198:36865): Couldn't send request to peer 4b7c382c9edd41babdd1a1afc904a813. Status: Network error: Client connection negotiation failed: client connection to 127.9.176.198:36865: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:47.767457 17499 raft_consensus.cc:1261] T 91e0e82526c0489a8908e250bf3c851f P 311f765f59754bdcb4eadc04e14bd4ea [term 1 LEARNER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
W20240502 01:54:47.778741 17435 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.176.194:45617, user_credentials={real_user=slave}} blocked reactor thread for 142105us
I20240502 01:54:47.754521 17763 raft_consensus.cc:2932] T a5e03cf44eab41cc842342a138c04067 P 84d3a4b174c34e02aedab1e620731359 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 272b2179c1b4476090a89880df2a8baa (127.9.176.199) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } }
W20240502 01:54:47.829797 17700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.176.196:46381, user_credentials={real_user=slave}} blocked reactor thread for 48636.2us
I20240502 01:54:47.878777 17746 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:54:47.932816 17478 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:54:47.941660 22305 raft_consensus.cc:1050] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 84d3a4b174c34e02aedab1e620731359: attempting to promote NON_VOTER 272b2179c1b4476090a89880df2a8baa to VOTER
W20240502 01:54:47.975487 17432 connection.cc:535] client connection to 127.9.176.197:35011 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 01:54:48.107075 19718 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:48.188966 19718 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:54:47.983000 22657 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 01:54:47.976532 (+     0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.197:35011 (local address 127.9.176.193:38443)
0502 01:54:47.978076 (+  1544us) negotiation.cc:107] Waiting for socket to connect
0502 01:54:47.978098 (+    22us) client_negotiation.cc:174] Beginning negotiation
0502 01:54:47.978238 (+   140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 01:54:47.981756 (+  3518us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.197:35011: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":1408}
I20240502 01:54:47.841238 17352 catalog_manager.cc:5484] T a5e03cf44eab41cc842342a138c04067 P 84d3a4b174c34e02aedab1e620731359 reported cstate change: config changed from index 7 to 8, 272b2179c1b4476090a89880df2a8baa (127.9.176.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "da3584f429104d23a1a8899b35cc2138" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } }
W20240502 01:54:48.227373 17432 consensus_peers.cc:485] T 49ef898a43704b87ba077cd7b0485e9a P 311f765f59754bdcb4eadc04e14bd4ea -> Peer 29806d3d37354c169484f6db368f8798 (127.9.176.197:35011): Couldn't send request to peer 29806d3d37354c169484f6db368f8798. Status: Network error: Client connection negotiation failed: client connection to 127.9.176.197:35011: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:48.227538 17432 consensus_peers.cc:485] T a6b5231a18a5414dbd010d4d48f08dbd P 311f765f59754bdcb4eadc04e14bd4ea -> Peer 29806d3d37354c169484f6db368f8798 (127.9.176.197:35011): Couldn't send request to peer 29806d3d37354c169484f6db368f8798. Status: Network error: Client connection negotiation failed: client connection to 127.9.176.197:35011: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:47.754858 19869 raft_consensus.cc:2932] T 38a60d7db47149ff90444e5276fc8935 P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 29806d3d37354c169484f6db368f8798 (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } }
I20240502 01:54:47.756932 19872 raft_consensus.cc:1261] T 91e0e82526c0489a8908e250bf3c851f P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Refusing update from remote peer 1b50a031f2784ba498b3e52e16916205: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 01:54:47.753613 22086 raft_consensus.cc:2932] T 38a60d7db47149ff90444e5276fc8935 P 1b50a031f2784ba498b3e52e16916205 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 29806d3d37354c169484f6db368f8798 (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } }
I20240502 01:54:47.790989 22600 consensus_queue.cc:1035] T 91e0e82526c0489a8908e250bf3c851f P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.006s
I20240502 01:54:48.230544 22600 consensus_queue.cc:1035] T 91e0e82526c0489a8908e250bf3c851f P 1b50a031f2784ba498b3e52e16916205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240502 01:54:47.795974 17565 consensus_peers.cc:485] T b40f968655774ce887dff8b7f4651501 P 1b50a031f2784ba498b3e52e16916205 -> Peer 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198:36865): Couldn't send request to peer 4b7c382c9edd41babdd1a1afc904a813. Status: Network error: Client connection negotiation failed: client connection to 127.9.176.198:36865: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:48.231256 17565 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.176.198:36865, user_credentials={real_user=slave}} blocked reactor thread for 435255us
I20240502 01:54:48.182686 17478 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:54:48.232717 17565 connection.cc:535] client connection to 127.9.176.197:35011 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:54:48.242187 17565 consensus_peers.cc:462] T df16e178e04040699201f325ce9e2979 P 1b50a031f2784ba498b3e52e16916205 -> Peer 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198:36865): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.9.176.198:36865: connect: Connection refused (error 111)
W20240502 01:54:48.277029 17565 consensus_peers.cc:485] T 974f67157aa04d3e8b49b264cfa104e8 P 1b50a031f2784ba498b3e52e16916205 -> Peer 29806d3d37354c169484f6db368f8798 (127.9.176.197:35011): Couldn't send request to peer 29806d3d37354c169484f6db368f8798. Status: Network error: Client connection negotiation failed: client connection to 127.9.176.197:35011: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:48.277297 17565 consensus_peers.cc:462] T b40f968655774ce887dff8b7f4651501 P 1b50a031f2784ba498b3e52e16916205 -> Peer 29806d3d37354c169484f6db368f8798 (127.9.176.197:35011): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.9.176.197:35011: connect: Connection refused (error 111)
I20240502 01:54:48.242431 17498 raft_consensus.cc:2932] T 38a60d7db47149ff90444e5276fc8935 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 29806d3d37354c169484f6db368f8798 (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } }
I20240502 01:54:47.752574 22539 raft_consensus.cc:2932] T a5e03cf44eab41cc842342a138c04067 P da3584f429104d23a1a8899b35cc2138 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 272b2179c1b4476090a89880df2a8baa (127.9.176.199) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } }
I20240502 01:54:47.942808 22305 consensus_queue.cc:237] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } }
W20240502 01:54:47.977526 17700 connection.cc:535] client connection to 127.9.176.197:35011 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 01:54:48.340099 17497 raft_consensus.cc:2932] T 91e0e82526c0489a8908e250bf3c851f P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { promote: false } } }
W20240502 01:54:47.975725 17832 connection.cc:535] client connection to 127.9.176.197:35011 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 01:54:48.487075 17880 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:54:48.488154 17880 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:54:48.496466 17478 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:54:48.505671 17832 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.176.197:35011, user_credentials={real_user=slave}} blocked reactor thread for 45649.5us
I20240502 01:54:48.507750 19735 raft_consensus.cc:2932] T a5e03cf44eab41cc842342a138c04067 P 272b2179c1b4476090a89880df2a8baa [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 272b2179c1b4476090a89880df2a8baa (127.9.176.199) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { replace: true } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } }
W20240502 01:54:48.529404 17832 consensus_peers.cc:485] T a5e03cf44eab41cc842342a138c04067 P da3584f429104d23a1a8899b35cc2138 -> Peer 29806d3d37354c169484f6db368f8798 (127.9.176.197:35011): Couldn't send request to peer 29806d3d37354c169484f6db368f8798. Status: Network error: Client connection negotiation failed: client connection to 127.9.176.197:35011: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:48.777794 17832 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.176.197:35011, user_credentials={real_user=slave}} blocked reactor thread for 248332us
I20240502 01:54:48.295858 19852 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:48.584671 17497 raft_consensus.cc:1261] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240502 01:54:48.277690 17565 consensus_peers.cc:485] T 91e0e82526c0489a8908e250bf3c851f P 1b50a031f2784ba498b3e52e16916205 -> Peer 29806d3d37354c169484f6db368f8798 (127.9.176.197:35011): Couldn't send request to peer 29806d3d37354c169484f6db368f8798. Status: Network error: Client connection negotiation failed: client connection to 127.9.176.197:35011: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:48.780297 17565 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.176.197:35011, user_credentials={real_user=slave}} blocked reactor thread for 502580us
W20240502 01:54:48.780558 17565 consensus_peers.cc:485] T 38a60d7db47149ff90444e5276fc8935 P 1b50a031f2784ba498b3e52e16916205 -> Peer 29806d3d37354c169484f6db368f8798 (127.9.176.197:35011): Couldn't send request to peer 29806d3d37354c169484f6db368f8798. Status: Network error: Client connection negotiation failed: client connection to 127.9.176.197:35011: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:48.780702 17565 consensus_peers.cc:328] unable to start election on peer 29806d3d37354c169484f6db368f8798: Network error: Client connection negotiation failed: client connection to 127.9.176.197:35011: connect: Connection refused (error 111)
W20240502 01:54:48.780985 17565 consensus_peers.cc:485] T 2c78951695284b1aaab411c0a041c28a P 1b50a031f2784ba498b3e52e16916205 -> Peer 29806d3d37354c169484f6db368f8798 (127.9.176.197:35011): Couldn't send request to peer 29806d3d37354c169484f6db368f8798. Status: Network error: Client connection negotiation failed: client connection to 127.9.176.197:35011: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:48.558892 19919 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
       Statistic       |   Value
-----------------------+-----------
 Minimum Replica Count | 0
 Maximum Replica Count | 23
 Average Replica Count | 11.250000

Per-table replica distribution summary:
 Replica Skew |  Value
--------------+----------
 Minimum      | 3
 Maximum      | 3
 Average      | 3.000000


stderr: W20240502 01:54:27.026944 19927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:54:27.027558 19927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:54:27.043196 19927 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 01:54:30.474646 19927 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240502 01:54:32.457602 19927 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table bf67e64a43cd4367a7420c6bf1793faa: could not find any suitable replica to move from server 311f765f59754bdcb4eadc04e14bd4ea to server 94931f4b51c9493cb3d4264ce99f5e76
W20240502 01:54:32.457898 19927 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 60aa7e7a81564c3a888cc9b68700c9ad: could not find any suitable replica to move from server 1b50a031f2784ba498b3e52e16916205 to server 272b2179c1b4476090a89880df2a8baa
W20240502 01:54:32.458238 19927 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table bf67e64a43cd4367a7420c6bf1793faa: could not find any suitable replica to move from server da3584f429104d23a1a8899b35cc2138 to server 4b7c382c9edd41babdd1a1afc904a813
W20240502 01:54:32.458473 19927 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table d49bf19a92cf45ceadf02d01dbdae6ad: could not find any suitable replica to move from server 311f765f59754bdcb4eadc04e14bd4ea to server 272b2179c1b4476090a89880df2a8baa
W20240502 01:54:32.458591 19927 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c5e1575e337a4256a90ce5b685b3ff28: could not find any suitable replica to move from server 84d3a4b174c34e02aedab1e620731359 to server 272b2179c1b4476090a89880df2a8baa
W20240502 01:54:32.458756 19927 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c516e7439e734f02a0947c3680cf79e2: could not find any suitable replica to move from server 311f765f59754bdcb4eadc04e14bd4ea to server 94931f4b51c9493cb3d4264ce99f5e76
W20240502 01:54:32.458860 19927 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 14b25b3f4ede4b65870815b014d90449: could not find any suitable replica to move from server 1b50a031f2784ba498b3e52e16916205 to server 29806d3d37354c169484f6db368f8798
W20240502 01:54:32.459017 19927 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table bf67e64a43cd4367a7420c6bf1793faa: could not find any suitable replica to move from server 1b50a031f2784ba498b3e52e16916205 to server 272b2179c1b4476090a89880df2a8baa
W20240502 01:54:32.459113 19927 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 952912d7ec8d493bbb8985fffe9aa918: could not find any suitable replica to move from server 84d3a4b174c34e02aedab1e620731359 to server 94931f4b51c9493cb3d4264ce99f5e76
W20240502 01:54:32.459234 19927 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 73ba62f6c85943bf85dfaaf459d924f0: could not find any suitable replica to move from server da3584f429104d23a1a8899b35cc2138 to server 4b7c382c9edd41babdd1a1afc904a813
W20240502 01:54:32.459335 19927 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 60aa7e7a81564c3a888cc9b68700c9ad: could not find any suitable replica to move from server 1b50a031f2784ba498b3e52e16916205 to server 94931f4b51c9493cb3d4264ce99f5e76
W20240502 01:54:32.459522 19927 rebalancer_tool.cc:1208] No rI20240502 01:54:48.338037 22600 raft_consensus.cc:2932] T 91e0e82526c0489a8908e250bf3c851f P 1b50a031f2784ba498b3e52e16916205 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { promote: false } } }
W20240502 01:54:48.634155 17553 kernel_stack_watchdog.cc:198] Thread 17565 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:308 for 295ms:
Kernel stack:
(could not read kernel stack)
User stack:
    @     0x7f17b71ae980  (unknown) at ??:0
    @     0x7f17affd0255  __write at ??:0
    @     0x7f17aff4b28d  _IO_file_write at ??:0
    @     0x7f17aff4bbff  _IO_file_xsputn at ??:0
    @     0x7f17aff3fa57  fwrite at ??:0
    @           0x8793e7  __interceptor_fwrite.part.56 at ??:?
    @     0x7f17b11328a0  google::LogMessage::SendToLog() at ??:0
    @     0x7f17b112e7cc  google::LogMessage::Flush() at ??:0
    @     0x7f17b112ea11  google::LogMessage::~LogMessage() at ??:0
    @     0x7f17bf56b9ee  kudu::consensus::Peer::ProcessResponseErrorUnlocked() at ??:0
    @     0x7f17bf56a69f  kudu::consensus::Peer::ProcessResponse() at ??:0
    @     0x7f17bf571061  kudu::consensus::Peer::SendNextRequest()::$_3::operator()() at ??:0
    @     0x7f17bf570e6e  std::_Function_handler<>::_M_invoke() at ??:0
    @           0xa60bcd  std::function<>::operator()() at ??:?
    @     0x7f17baa5a4e0  kudu::rpc::OutboundCall::CallCallback() at ??:0
    @     0x7f17baa5b365  kudu::rpc::OutboundCall::SetFailed() at ??:0
W20240502 01:54:48.833894 17565 consensus_peers.cc:462] T df16e178e04040699201f325ce9e2979 P 1b50a031f2784ba498b3e52e16916205 -> Peer 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198:36865): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.9.176.198:36865: connect: Connection refused (error 111)
W20240502 01:54:48.895231 17565 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.StartTabletCopy -> {remote=127.9.176.198:36865, user_credentials={real_user=slave}} blocked reactor thread for 61319.6us
W20240502 01:54:48.895656 17565 consensus_peers.cc:462] T b40f968655774ce887dff8b7f4651501 P 1b50a031f2784ba498b3e52e16916205 -> Peer 29806d3d37354c169484f6db368f8798 (127.9.176.197:35011): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.9.176.197:35011: connect: Connection refused (error 111)
eplica could be moved this iteration: Not found: table 40a6be2c1a2744428cea56b78225e44e: could not find any suitable replica to move from server 311f765f59754bdcb4eadc04e14bd4ea to server 29806d3d37354c169484f6db368f8798
W20240502 01:54:32.459721 19927 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1bd69df0513c4c4db4dd6d6526efee4b: could not find any suitable replica to move from server 1b50a031f2784ba498b3e52e16916205 to server 272b2179c1b4476090a89880df2a8baa
W20240502 01:54:32.459848 19927 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table bf67e64a43cd4367a7420c6bf1793faa: could not find any suitable replica to move from server 311f765f59754bdcb4eadc04e14bd4ea to server 272b2179c1b4476090a89880df2a8baa
W20240502 01:54:32.459975 19927 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 952912d7ec8d493bbb8985fffe9aa918: could not find any suitable replica to move from server da3584f429104d23a1a8899b35cc2138 to server 94931f4b51c9493cb3d4264ce99f5e76
W20240502 01:54:32.460434 19927 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 73ba62f6c85943bf85dfaaf459d924f0: could not find any suitable replica to move from server 1b50a031f2784ba498b3e52e16916205 to server 29806d3d37354c169484f6db368f8798
I20240502 01:54:32.720506 19927 rebalancer_tool.cc:1074] tablet 3bdd49939e994240a6d15dbd11559ff4: '84d3a4b174c34e02aedab1e620731359' -> '94931f4b51c9493cb3d4264ce99f5e76' move scheduled
I20240502 01:54:33.000102 19927 rebalancer_tool.cc:1074] tablet c91f74f46b04425aa5393515cb9f781c: 'da3584f429104d23a1a8899b35cc2138' -> '29806d3d37354c169484f6db368f8798' move scheduled
I20240502 01:54:33.302492 19927 rebalancer_tool.cc:1074] tablet 768e7cbf63a94d3a815ac77a4d0acf41: '1b50a031f2784ba498b3e52e16916205' -> '4b7c382c9edd41babdd1a1afc904a813' move scheduled
I20240502 01:54:33.698554 19927 rebalancer_tool.cc:1074] tablet df16e178e04040699201f325ce9e2979: '311f765f59754bdcb4eadc04e14bd4ea' -> '272b2179c1b4476090a89880df2a8baa' move scheduled
I20240502 01:54:34.371279 19927 rebalancer_tool.cc:1074] tablet 38a60d7db47149ff90444e5276fc8935: '84d3a4b174c34e02aedab1e620731359' -> '94931f4b51c9493cb3d4264ce99f5e76' move scheduled
I20240502 01:54:34.540874 19927 rebalancer_tool.cc:1074] tablet 974f67157aa04d3e8b49b264cfa104e8: 'da3584f429104d23a1a8899b35cc2138' -> '29806d3d37354c169484f6db368f8798' move scheduled
I20240502 01:54:35.052577 19927 rebalancer_tool.cc:1074] tablet 88a137bcfc76410583f910b377d1be7f: '1b50a031f2784ba498b3e52e16916205' -> '4b7c382c9edd41babdd1a1afc904a813' move scheduled
I20240502 01:54:35.386837 19927 rebalancer_tool.cc:1074] tablet f07e9b433e554c85b3c25b7e708f8b2f: '311f765f59754bdcb4eadc04e14bd4ea' -> '94931f4b51c9493cb3d4264ce99f5e76' move scheduled
W20240502 01:54:35.851521 21102 sasl_common.cc:323] Time spent creating new SASL kudu client: real 0.278s	user 0.000s	sys 0.000s
I20240502 01:54:35.877002 19927 rebalancer_tool.cc:1084] tablet 37eda67598fe491bbb29aa62b2c4bec7: '84d3a4b174c34e02aedab1e620731359' -> '272b2179c1b4476090a89880df2a8baa' move ignored: Not found: Peer with uuid 84d3a4b174c34e02aedab1e620731359 not found in consensus config
I20240502 01:54:35.877199 19927 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 01:54:39.958683 19927 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c5e1575e337a4256a90ce5b685b3ff28: could not find any suitable replica to move from server 311f765f59754bdcb4eadc04e14bd4ea to server 272b2179c1b4476090a89880df2a8baa
W20240502 01:54:39.959025 19927 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table bf67e64a43cd4367a7420c6bf1793faa: could not find any suitable replica to move from server 1b50a031f2784ba498b3e52e16916205 to server da3584f429104d23a1a8899b35cc2138
W20240502 01:54:39.959280 19927 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 14b25b3f4ede4b65870815b014d90449: could not find any suitable replica to move from server 84d3a4b174c34e02aedab1e620731359 to server 272b2179c1b4476090a89880df2a8baa
W20240502 01:54:39.959520 19927 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table bf67e64a43cd4367a7420c6bf1793faa: could not find any suitable replica to move from server 311f765f59754bdcb4eadc04e14bd4ea to server 94931f4b51c9493cb3d4264ce99f5e76
W20240502 01:54:39.959740 19927 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 73ba62f6c85943bf85dfaaf459d924f0: could not find any suitable replica to move from server 1b50a031f2784ba498b3e52e16916205 to server 94931f4b51c9493cb3d4264ce99f5e76
W20240502 01:54:39.959895 19927 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 60aa7e7a81564c3a888cc9b68700c9ad: could not find any suitable replica to move from server 84d3a4b174c34e02aedab1e620731359 to server 4b7c382c9edd41babdd1a1afc904a813
W20240502 01:54:39.959997 19927 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c5e1575e337a4256a90ce5b685b3ff28: could not find any suitable replica to move from server da3584f429104d23a1a8899b35cc2138 to server 272b2179c1b4476090a89880df2a8baa
W20240502 01:54:39.960132 19927 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table d49bf19a92cf45ceadf02d01dbdae6ad: could not find any suitable replica to move from server 311f765f59754bdcb4eadc04e14bd4ea to server 94931f4b51c9493cb3d4264ce99f5e76
I20240502 01:54:40.065757 19927 rebalancer_tool.cc:1074] tablet 2c78951695284b1aaab411c0a041c28a: '1b50a031f2784ba498b3e52e16916205' -> '' move scheduled
I20240502 01:54:40.844033 19927 rebalancer_tool.cc:1074] tablet 6da59c0626c64a5988e39a831c42c268: '84d3a4b174c34e02aedab1e620731359' -> '272b2179c1b4476090a89880df2a8baa' move scheduled
I20240502 01:54:41.153167 19927 rebalancer_tool.cc:1074] tablet 3b3fa060e8d64911bec810ec07a358d2: '311f765f59754bdcb4eadc04e14bd4ea' -> '4b7c382c9edd41babdd1a1afc904a813' move scheduled
I20240502 01:54:41.487135 19927 rebalancer_tool.cc:1074] tablet 2045e08a200148c69326351f38fd484d: '29806d3d37354c169484f6db368f8798' -> 'da3584f429104d23a1a8899b35cc2138' move scheduled
I20240502 01:54:41.789754 19927 rebalancer_tool.cc:1074] tablet 60ff2e50b1d74e0c8e0890b87f9ac8ba: '94931f4b51c9493cb3d4264ce99f5e76' -> '84d3a4b174c34e02aedab1e620731359' move scheduled
I20240502 01:54:42.272559 19927 rebalancer_tool.cc:1084] tablet 49ef898a43704b87ba077cd7b0485e9a: '1b50a031f2784ba498b3e52e16916205' -> '4b7c382c9edd41babdd1a1afc904a813' move ignored: Not found: Peer with uuid 1b50a031f2784ba498b3e52e16916205 not found in consensus config
I20240502 01:54:42.272696 19927 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 01:54:45.299403 19927 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table d49bf19a92cf45ceadf02d01dbdae6ad: could not find any suitable replica to move from server 311f765f59754bdcb4eadc04e14bd4ea to server 94931f4b51c9493cb3d4264ce99f5e76
W20240502 01:54:45.299645 19927 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 952912d7ec8d493bbb8985fffe9aa918: could not find any suitable replica to move from server da3584f429104d23a1a8899b35cc2138 to server 29806d3d37354c169484f6db368f8798
W20240502 01:54:45.299861 19927 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 60aa7e7a81564c3a888cc9b68700c9ad: could not find any suitable replica to move from server 1b50a031f2784ba498b3e52e16916205 to server 272b2179c1b4476090a89880df2a8baa
W20240502 01:54:45.299975 19927 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 40a6be2c1a2744428cea56b78225e44e: could not find any suitable replica to move from server 84d3a4b174c34e02aedab1e620731359 to server 94931f4b51c9493cb3d4264ce99f5e76
I20240502 01:54:45.443483 19927 rebalancer_tool.cc:1074] tablet a5e03cf44eab41cc842342a138c04067: 'da3584f429104d23a1a8899b35cc2138' -> '272b2179c1b4476090a89880df2a8baa' move scheduled
I20240502 01:54:45.686905 19927 rebalancer_tool.cc:1074] tablet 91e0e82526c0489a8908e250bf3c851f: '1b50a031f2784ba498b3e52e16916205' -> '311f765f59754bdcb4eadc04e14bd4ea' move scheduled
I20240502 01:54:45.838078 19927 rebalancer_tool.cc:1074] tablet 17365abbb6c1430aa278719eb8e4a648: '4b7c382c9edd41babdd1a1afc904a813' -> '29806d3d37354c169484f6db368f8798' move scheduled
I20240502 01:54:45.917001 19927 rebalancer_tool.cc:1084] tablet 3ca5c4988ff74e3ca8d7a5614bcb93e2: '84d3a4b174c34e02aedab1e620731359' -> '272b2179c1b4476090a89880df2a8baa' move ignored: Illegal state: tablet 3ca5c4988ff74e3ca8d7a5614bcb93e2: replica 272b2179c1b4476090a89880df2a8baa is already present
I20240502 01:54:45.917191 19927 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 01:54:47.054211 22621 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 01:54:47.006687 (+     0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.198:36865 (local address 127.0.0.1:43894)
0502 01:54:47.006943 (+   256us) negotiation.cc:107] Waiting for socket to connect
0502 01:54:47.006965 (+    22us) client_negotiation.cc:174] Beginning negotiation
0502 01:54:47.007116 (+   151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 01:54:47.052656 (+ 45540us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.198:36865: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":124}
W20240502 01:54:47.990677 22753 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 01:54:47.975346 (+     0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.197:35011 (local address 127.0.0.1:33230)
0502 01:54:47.990311 (+ 14965us) negotiation.cc:107] Waiting for socket to connect
0502 01:54:47.990430 (+   119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.197:35011: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":14832,"threads_started":1}
Illegal state: tablet server 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198:36865): unacceptable health status UNAVAILABLE

W20240502 01:54:48.531260 17700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.176.197:35011, user_credentials={real_user=slave}} blocked reactor thread for 75463.1us
I20240502 01:54:48.582895 17746 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:54:48.583178 17745 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:54:48.292937 17348 catalog_manager.cc:5484] T 38a60d7db47149ff90444e5276fc8935 P 94931f4b51c9493cb3d4264ce99f5e76 reported cstate change: config changed from index 9 to 10, 29806d3d37354c169484f6db368f8798 (127.9.176.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1b50a031f2784ba498b3e52e16916205" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } }
W20240502 01:54:48.719679 17821 kernel_stack_watchdog.cc:198] Thread 17832 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:308 for 135ms:
Kernel stack:
(could not read kernel stack)
User stack:
    @     0x7f7441631980  (unknown) at ??:0
    @     0x7f743a453255  __write at ??:0
    @     0x7f743a3ce28d  _IO_file_write at ??:0
    @     0x7f743a3cebff  _IO_file_xsputn at ??:0
    @     0x7f743a3c2a57  fwrite at ??:0
    @           0x8793e7  __interceptor_fwrite.part.56 at ??:?
    @     0x7f743b5b58a0  google::LogMessage::SendToLog() at ??:0
    @     0x7f743b5b17cc  google::LogMessage::Flush() at ??:0
    @     0x7f743b5b1a11  google::LogMessage::~LogMessage() at ??:0
    @     0x7f74499ee9ee  kudu::consensus::Peer::ProcessResponseErrorUnlocked() at ??:0
    @     0x7f74499ed69f  kudu::consensus::Peer::ProcessResponse() at ??:0
    @     0x7f74499f4061  kudu::consensus::Peer::SendNextRequest()::$_3::operator()() at ??:0
    @     0x7f74499f3e6e  std::_Function_handler<>::_M_invoke() at ??:0
    @           0xa60bcd  std::function<>::operator()() at ??:?
    @     0x7f7444edd4e0  kudu::rpc::OutboundCall::CallCallback() at ??:0
    @     0x7f7444ede365  kudu::rpc::OutboundCall::SetFailed() at ??:0
I20240502 01:54:48.299294 19851 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:48.782737 22305 consensus_queue.cc:1035] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:48.893857 17747 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:54:48.823913 17880 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:54:48.330075 19850 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:48.795178 19870 raft_consensus.cc:2932] T 91e0e82526c0489a8908e250bf3c851f P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { promote: false } } }
I20240502 01:54:48.887452 19718 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:48.920214 19736 raft_consensus.cc:1261] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 272b2179c1b4476090a89880df2a8baa [term 1 LEARNER]: Refusing update from remote peer 84d3a4b174c34e02aedab1e620731359: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240502 01:54:48.920979 17565 consensus_peers.cc:462] T df16e178e04040699201f325ce9e2979 P 1b50a031f2784ba498b3e52e16916205 -> Peer 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198:36865): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.9.176.198:36865: connect: Connection refused (error 111)
W20240502 01:54:48.975492 17700 consensus_peers.cc:485] T c91f74f46b04425aa5393515cb9f781c P 84d3a4b174c34e02aedab1e620731359 -> Peer 29806d3d37354c169484f6db368f8798 (127.9.176.197:35011): Couldn't send request to peer 29806d3d37354c169484f6db368f8798. Status: Network error: Client connection negotiation failed: client connection to 127.9.176.197:35011: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:48.980114 17700 consensus_peers.cc:485] T 6da59c0626c64a5988e39a831c42c268 P 84d3a4b174c34e02aedab1e620731359 -> Peer 29806d3d37354c169484f6db368f8798 (127.9.176.197:35011): Couldn't send request to peer 29806d3d37354c169484f6db368f8798. Status: Network error: Client connection negotiation failed: client connection to 127.9.176.197:35011: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:48.989976 22305 consensus_queue.cc:1035] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 84d3a4b174c34e02aedab1e620731359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240502 01:54:48.993741 17700 consensus_peers.cc:485] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 84d3a4b174c34e02aedab1e620731359 -> Peer 29806d3d37354c169484f6db368f8798 (127.9.176.197:35011): Couldn't send request to peer 29806d3d37354c169484f6db368f8798. Status: Network error: Client connection negotiation failed: client connection to 127.9.176.197:35011: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:49.053956 17355 catalog_manager.cc:5484] T 91e0e82526c0489a8908e250bf3c851f P 94931f4b51c9493cb3d4264ce99f5e76 reported cstate change: config changed from index 10 to 11, 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1b50a031f2784ba498b3e52e16916205" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { promote: false } } }
I20240502 01:54:49.105383 22305 raft_consensus.cc:2932] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 84d3a4b174c34e02aedab1e620731359 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 272b2179c1b4476090a89880df2a8baa (127.9.176.199) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } }
I20240502 01:54:49.109753 17613 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:54:49.111053 17613 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:54:49.111418 17612 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:54:49.116740 17352 catalog_manager.cc:5484] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 84d3a4b174c34e02aedab1e620731359 reported cstate change: config changed from index 7 to 8, 272b2179c1b4476090a89880df2a8baa (127.9.176.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "84d3a4b174c34e02aedab1e620731359" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:49.125088 17478 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:54:49.126346 17435 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.176.194:45617, user_credentials={real_user=slave}} blocked reactor thread for 115440us
I20240502 01:54:49.138767 19736 raft_consensus.cc:2932] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 272b2179c1b4476090a89880df2a8baa [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 272b2179c1b4476090a89880df2a8baa (127.9.176.199) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } }
I20240502 01:54:49.151520 19850 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:49.160909 17497 raft_consensus.cc:2932] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 311f765f59754bdcb4eadc04e14bd4ea [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 272b2179c1b4476090a89880df2a8baa (127.9.176.199) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } attrs { replace: true } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } }
I20240502 01:54:49.179556 17880 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:54:49.278306 17565 consensus_peers.cc:462] T b40f968655774ce887dff8b7f4651501 P 1b50a031f2784ba498b3e52e16916205 -> Peer 29806d3d37354c169484f6db368f8798 (127.9.176.197:35011): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.9.176.197:35011: connect: Connection refused (error 111)
W20240502 01:54:49.446080 17432 consensus_peers.cc:485] T fadcc8fe6ae047c1b142514aad8bebba P 311f765f59754bdcb4eadc04e14bd4ea -> Peer 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198:36865): Couldn't send request to peer 4b7c382c9edd41babdd1a1afc904a813. Status: Network error: Client connection negotiation failed: client connection to 127.9.176.198:36865: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:54:49.482453 17565 consensus_peers.cc:462] T df16e178e04040699201f325ce9e2979 P 1b50a031f2784ba498b3e52e16916205 -> Peer 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198:36865): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.9.176.198:36865: connect: Connection refused (error 111)
I20240502 01:54:49.606338 19920 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
       Statistic       |   Value
-----------------------+-----------
 Minimum Replica Count | 0
 Maximum Replica Count | 23
 Average Replica Count | 11.250000

Per-table replica distribution summary:
 Replica Skew |  Value
--------------+----------
 Minimum      | 3
 Maximum      | 3
 Average      | 3.000000


stderr: W20240502 01:54:27.095912 19928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:54:27.096441 19928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:54:27.108915 19928 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 01:54:29.988838 19928 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240502 01:54:31.881719 19928 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 952912d7ec8d493bbb8985fffe9aa918: could not find any suitable replica to move from server 84d3a4b174c34e02aedab1e620731359 to server 29806d3d37354c169484f6db368f8798
W20240502 01:54:31.882136 19928 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table d49bf19a92cf45ceadf02d01dbdae6ad: could not find any suitable replica to move from server 311f765f59754bdcb4eadc04e14bd4ea to server 272b2179c1b4476090a89880df2a8baa
W20240502 01:54:31.882359 19928 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c5e1575e337a4256a90ce5b685b3ff28: could not find any suitable replica to move from server 84d3a4b174c34e02aedab1e620731359 to server 94931f4b51c9493cb3d4264ce99f5e76
W20240502 01:54:31.882534 19928 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 40a6be2c1a2744428cea56b78225e44e: could not find any suitable replica to move from server 311f765f59754bdcb4eadc04e14bd4ea to server 272b2179c1b4476090a89880df2a8baa
W20240502 01:54:31.882660 19928 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 14b25b3f4ede4b65870815b014d90449: could not find any suitable replica to move from server da3584f429104d23a1a8899b35cc2138 to server 29806d3d37354c169484f6db368f8798
W20240502 01:54:31.882931 19928 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table bf67e64a43cd4367a7420c6bf1793faa: could not find any suitable replica to move from server 311f765f59754bdcb4eadc04e14bd4ea to server 94931f4b51c9493cb3d4264ce99f5e76
W20240502 01:54:31.883071 19928 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 60aa7e7a81564c3a888cc9b68700c9ad: could not find any suitable replica to move from server da3584f429104d23a1a8899b35cc2138 to server 29806d3d37354c169484f6db368f8798
W20240502 01:54:31.883306 19928 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 73ba62f6c85943bf85dfaaf459d924f0: could not find any suitable replica to move from server 84d3a4b174c34e02aedab1e620731359 to server 29806d3d37354c169484f6db368f8798
W20240502 01:54:31.883481 19928 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c516e7439e734f02a0947c3680cf79e2: could not find any suitable replica to move from server 311f765f59754bdcb4eadc04e14bd4ea to server 272b2179c1b4476090a89880df2a8baa
W20240502 01:54:31.883793 19928 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 40a6be2c1a2744428cea56b78225e44e: could not find any suitable replica to move from server 84d3a4b174c34e02aedab1e620731359 to server 94931f4b51c9493cb3d4264ce99f5e76
W20240502 01:54:31.883968 19928 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1bd69df0513c4c4db4dd6d6526efee4b: could not find any suitable replica to move from server 311f765f59754bdcb4eadc04e14bd4ea to server 272b2179c1b4476090a89880df2a8baa
W20240502 01:54:31.884181 19928 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table bf67e64a43cd4367a7420c6bf1793faa: could not find any suitable replica to move from server da3584f429104d23a1a8899b35cc2138 to server 29806d3d37354c169484f6db368f8798
W20240502 01:54:31.884341 19928 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 60aa7e7a81564c3a888cc9b68700c9ad: could not find any suitable replica to move from server 1b50a031f2784ba498b3e52e16916205 to server 4b7c382c9edd41babdd1a1afc904a813
W20240502 01:54:31.884493 19928 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 952912d7ec8d493bbb8985fffe9aa918: could not find any suitable replica to move from server da3584f429104d23a1a8899b35cc2138 to server 94931f4b51c9493cb3d4264ce99f5e76
W20240502 01:54:31.884737 19928 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 73ba62f6c85943bf85dfaaf459d924f0: could not find any suitable replica to move from server 1b50a031f2784ba498b3e52e16916205 to server 272b2179c1b4476090a89880df2a8baa
I20240502 01:54:32.264536 19928 rebalancer_tool.cc:1074] tablet 3b3fa060e8d64911bec810ec07a358d2: 'da3584f429104d23a1a8899b35cc2138' -> '272b2179c1b4476090a89880df2a8baa' move scheduled
I20240502 01:54:32.468847 19928 rebalancer_tool.cc:1074] tablet 6da59c0626c64a5988e39a831c42c268: '311f765f59754bdcb4eadc04e14bd4ea' -> '29806d3d37354c169484f6db368f8798' move scheduled
I20240502 01:54:32.854974 19928 rebalancer_tool.cc:1074] tablet 05b67d9912f047bfb6a2bed90b82ae15: '1b50a031f2784ba498b3e52e16916205' -> '4b7c382c9edd41babdd1a1afc904a813' move scheduled
I20240502 01:54:33.208545 19928 rebalancer_tool.cc:1074] tablet 37eda67598fe491bbb29aa62b2c4bec7: '84d3a4b174c34e02aedab1e620731359' -> '94931f4b51c9493cb3d4264ce99f5e76' move scheduled
I20240502 01:54:33.533752 19928 rebalancer_tool.cc:1074] tablet b40f968655774ce887dff8b7f4651501: 'da3584f429104d23a1a8899b35cc2138' -> '272b2179c1b4476090a89880df2a8baa' move scheduled
I20240502 01:54:33.686511 19928 rebalancer_tool.cc:1084] tablet df16e178e04040699201f325ce9e2979: '311f765f59754bdcb4eadc04e14bd4ea' -> '29806d3d37354c169484f6db368f8798' move ignored: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } }.
  Pending config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "84d3a4b174c34e02aedab1e620731359" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37357 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: NON_VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: true } }
I20240502 01:54:33.686730 19928 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 01:54:37.648360 19928 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 73ba62f6c85943bf85dfaaf459d924f0: could not find any suitable replica to move from server 84d3a4b174c34e02aedab1e620731359 to server 29806d3d37354c169484f6db368f8798
W20240502 01:54:37.650261 19928 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 952912d7ec8d493bbb8985fffe9aa918: could not find any suitable replica to move from server 1b50a031f2784ba498b3e52e16916205 to server 272b2179c1b4476090a89880df2a8baa
W20240502 01:54:37.650462 19928 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table bf67e64a43cd4367a7420c6bf1793faa: could not find any suitable replica to move from server 1b50a031f2784ba498b3e52e16916205 to server 4b7c382c9edd41babdd1a1afc904a813
W20240502 01:54:37.651219 19928 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 73ba62f6c85943bf85dfaaf459d924f0: could not find any suitable replica to move from server 1b50a031f2784ba498b3e52e16916205 to server 29806d3d37354c169484f6db368f8798
W20240502 01:54:37.651319 19928 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1bd69df0513c4c4db4dd6d6526efee4b: could not find any suitable replica to move from server 84d3a4b174c34e02aedab1e620731359 to server 94931f4b51c9493cb3d4264ce99f5e76
W20240502 01:54:37.651458 19928 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c5e1575e337a4256a90ce5b685b3ff28: could not find any suitable replica to move from server 311f765f59754bdcb4eadc04e14bd4ea to server 29806d3d37354c169484f6db368f8798
W20240502 01:54:37.651657 19928 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table d49bf19a92cf45ceadf02d01dbdae6ad: could not find any suitable replica to move from server 1b50a031f2784ba498b3e52e16916205 to server 272b2179c1b4476090a89880df2a8baa
W20240502 01:54:37.651757 19928 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c516e7439e734f02a0947c3680cf79e2: could not find any suitable replica to move from server 84d3a4b174c34e02aedab1e620731359 to server 272b2179c1b4476090a89880df2a8baa
W20240502 01:54:37.652705 19928 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 14b25b3f4ede4b65870815b014d90449: could not find any suitable replica to move from server da3584f429104d23a1a8899b35cc2138 to server 29806d3d37354c169484f6db368f8798
W20240502 01:54:37.652817 19928 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c5e1575e337a4256a90ce5b685b3ff28: could not find any suitable replica to move from server 84d3a4b174c34e02aedab1e620731359 to server 4b7c382c9edd41babdd1a1afc904a813
W20240502 01:54:37.652899 19928 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 40a6be2c1a2744428cea56b78225e44e: could not find any suitable replica to move from server 84d3a4b174c34e02aedab1e620731359 to server 29806d3d37354c169484f6db368f8798
W20240502 01:54:37.652995 19928 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table d49bf19a92cf45ceadf02d01dbdae6ad: could not find any suitable replica to move from server da3584f429104d23a1a8899b35cc2138 to server 4b7c382c9edd41babdd1a1afc904a813
W20240502 01:54:37.653066 19928 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 14b25b3f4ede4b65870815b014d90449: could not find any suitable replica to move from server 311f765f59754bdcb4eadc04e14bd4ea to server 272b2179c1b4476090a89880df2a8baa
W20240502 01:54:37.653138 19928 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 952912d7ec8d493bbb8985fffe9aa918: could not find any suitable replica to move from server 1b50a031f2784ba498b3e52e16916205 to server 94931f4b51c9493cb3d4264ce99f5e76
W20240502 01:54:37.653206 19928 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 73ba62f6c85943bf85dfaaf459d924f0: could not find any suitable replica to move from server 84d3a4b174c34e02aedab1e620731359 to server 94931f4b51c9493cb3d4264ce99f5e76
W20240502 01:54:37.653299 19928 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 60aa7e7a81564c3a888cc9b68700c9ad: could not find any suitable replica to move from server da3584f429104d23a1a8899b35cc2138 to server 4b7c382c9edd41babdd1a1afc904a813
W20240502 01:54:37.653414 19928 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1bd69df0513c4c4db4dd6d6526efee4b: could not find any suitable replica to move from server 1b50a031f2784ba498b3e52e16916205 to server 272b2179c1b4476090a89880df2a8baa
I20240502 01:54:38.198312 19928 rebalancer_tool.cc:1074] tablet 91e0e82526c0489a8908e250bf3c851f: '311f765f59754bdcb4eadc04e14bd4ea' -> '94931f4b51c9493cb3d4264ce99f5e76' move scheduled
I20240502 01:54:38.636838 19928 rebalancer_tool.cc:1074] tablet 5f3f3e5f772b4566b69236fd87d71a38: '84d3a4b174c34e02aedab1e620731359' -> '4b7c382c9edd41babdd1a1afc904a813' move scheduled
I20240502 01:54:39.151741 19928 rebalancer_tool.cc:1084] tablet a6b5231a18a5414dbd010d4d48f08dbd: 'da3584f429104d23a1a8899b35cc2138' -> '29806d3d37354c169484f6db368f8798' move ignored: Not found: Peer with uuid da3584f429104d23a1a8899b35cc2138 not found in consensus config
I20240502 01:54:39.152583 19928 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 01:54:42.329213 19928 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table bf67e64a43cd4367a7420c6bf1793faa: could not find any suitable replica to move from server 1b50a031f2784ba498b3e52e16916205 to server da3584f429104d23a1a8899b35cc2138
W20240502 01:54:42.330929 19928 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 14b25b3f4ede4b65870815b014d90449: could not find any suitable replica to move from server 84d3a4b174c34e02aedab1e620731359 to server 272b2179c1b4476090a89880df2a8baa
W20240502 01:54:42.332020 19928 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1bd69df0513c4c4db4dd6d6526efee4b: could not find any suitable replica to move from server 1b50a031f2784ba498b3e52e16916205 to server 94931f4b51c9493cb3d4264ce99f5e76
W20240502 01:54:42.333132 19928 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table bf67e64a43cd4367a7420c6bf1793faa: could not find any suitable replica to move from server 311f765f59754bdcb4eadc04e14bd4ea to server 4b7c382c9edd41babdd1a1afc904a813
I20240502 01:54:42.665222 19928 rebalancer_tool.cc:1084] tablet 49ef898a43704b87ba077cd7b0485e9a: '1b50a031f2784ba498b3e52e16916205' -> '' move ignored: Not found: Peer with uuid 1b50a031f2784ba498b3e52e16916205 not found in consensus config
I20240502 01:54:42.665402 19928 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 01:54:45.478484 19928 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table bf67e64a43cd4367a7420c6bf1793faa: could not find any suitable replica to move from server 1b50a031f2784ba498b3e52e16916205 to server 311f765f59754bdcb4eadc04e14bd4ea
I20240502 01:54:45.956439 19928 rebalancer_tool.cc:1074] tablet 7ce9d4281c1b46258493feb39e8d9ab2: '311f765f59754bdcb4eadc04e14bd4ea' -> '' move scheduled
I20240502 01:54:46.002498 19928 rebalancer_tool.cc:1084] tablet 38a60d7db47149ff90444e5276fc8935: '1b50a031f2784ba498b3e52e16916205' -> '29806d3d37354c169484f6db368f8798' move ignored: Illegal state: tablet 38a60d7db47149ff90444e5276fc8935: replica 29806d3d37354c169484f6db368f8798 is already present
I20240502 01:54:46.002681 19928 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 01:54:47.979270 22707 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 01:54:47.680637 (+     0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.197:35011 (local address 127.0.0.1:33228)
0502 01:54:47.680894 (+   257us) negotiation.cc:107] Waiting for socket to connect
0502 01:54:47.680913 (+    19us) client_negotiation.cc:174] Beginning negotiation
0502 01:54:47.681027 (+   114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 01:54:47.977772 (+296745us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.197:35011: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":132}
Illegal state: tablet server 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198:36865): unacceptable health status UNAVAILABLE

I20240502 01:54:49.630827 19921 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
       Statistic       |   Value
-----------------------+-----------
 Minimum Replica Count | 0
 Maximum Replica Count | 23
 Average Replica Count | 11.250000

Per-table replica distribution summary:
 Replica Skew |  Value
--------------+----------
 Minimum      | 3
 Maximum      | 3
 Average      | 3.000000


stderr: W20240502 01:54:27.573995 19929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:54:27.574571 19929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:54:27.586359 19929 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20240502 01:54:28.869366 20138 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTableSchema -> {remote=127.9.176.254:46743, user_credentials={real_user=slave}} blocked reactor thread for 47453.6us
I20240502 01:54:31.837523 19929 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240502 01:54:34.988070 19929 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table bf67e64a43cd4367a7420c6bf1793faa: could not find any suitable replica to move from server 311f765f59754bdcb4eadc04e14bd4ea to server 4b7c382c9edd41babdd1a1afc904a813
W20240502 01:54:34.988837 19929 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table bf67e64a43cd4367a7420c6bf1793faa: could not find any suitable replica to move from server da3584f429104d23a1a8899b35cc2138 to server 94931f4b51c9493cb3d4264ce99f5e76
W20240502 01:54:34.989180 19929 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 952912d7ec8d493bbb8985fffe9aa918: could not find any suitable replica to move from server 84d3a4b174c34e02aedab1e620731359 to server 94931f4b51c9493cb3d4264ce99f5e76
W20240502 01:54:34.989540 19929 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 73ba62f6c85943bf85dfaaf459d924f0: could not find any suitable replica to move from server 84d3a4b174c34e02aedab1e620731359 to server 29806d3d37354c169484f6db368f8798
W20240502 01:54:34.989709 19929 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 60aa7e7a81564c3a888cc9b68700c9ad: could not find any suitable replica to move from server 1b50a031f2784ba498b3e52e16916205 to server 94931f4b51c9493cb3d4264ce99f5e76
W20240502 01:54:34.989867 19929 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 40a6be2c1a2744428cea56b78225e44e: could not find any suitable replica to move from server 84d3a4b174c34e02aedab1e620731359 to server 29806d3d37354c169484f6db368f8798
W20240502 01:54:34.989981 19929 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 14b25b3f4ede4b65870815b014d90449: could not find any suitable replica to move from server 84d3a4b174c34e02aedab1e620731359 to server 272b2179c1b4476090a89880df2a8baa
W20240502 01:54:34.990218 19929 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table d49bf19a92cf45ceadf02d01dbdae6ad: could not find any suitable replica to move from server 311f765f59754bdcb4eadc04e14bd4ea to server 29806d3d37354c169484f6db368f8798
W20240502 01:54:34.990342 19929 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 73ba62f6c85943bf85dfaaf459d924f0: could not find any suitable replica to move from server 1b50a031f2784ba498b3e52e16916205 to server 94931f4b51c9493cb3d4264ce99f5e76
W20240502 01:54:34.990483 19929 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 40a6be2c1a2744428cea56b78225e44e: could not find any suitable replica to move from server 311f765f59754bdcb4eadc04e14bd4ea to server 4b7c382c9edd41babdd1a1afc904a813
W20240502 01:54:34.990672 19929 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1bd69df0513c4c4db4dd6d6526efee4b: could not find any suitable replica to move from server 84d3a4b174c34e02aedab1e620731359 to server 272b2179c1b4476090a89880df2a8baa
W20240502 01:54:34.990773 19929 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table bf67e64a43cd4367a7420c6bf1793faa: could not find any suitable replica to move from server da3584f429104d23a1a8899b35cc2138 to server 272b2179c1b4476090a89880df2a8baa
W20240502 01:54:34.990857 19929 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 952912d7ec8d493bbb8985fffe9aa918: could not find any suitable replica to move from server da3584f429104d23a1a8899b35cc2138 to server 272b2179c1b4476090a89880df2a8baa
W20240502 01:54:34.990994 19929 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table d49bf19a92cf45ceadf02d01dbdae6ad: could not find any suitable replica to move from server 1b50a031f2784ba498b3e52e16916205 to server 94931f4b51c9493cb3d4264ce99f5e76
W20240502 01:54:34.991112 19929 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c516e7439e734f02a0947c3680cf79e2: could not find any suitable replica to move from server da3584f429104d23a1a8899b35cc2138 to server 4b7c382c9edd41babdd1a1afc904a813
W20240502 01:54:34.991290 19929 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 40a6be2c1a2744428cea56b78225e44e: could not find any suitable replica to move from server 1b50a031f2784ba498b3e52e16916205 to server 29806d3d37354c169484f6db368f8798
I20240502 01:54:35.611270 19929 rebalancer_tool.cc:1074] tablet 60ff2e50b1d74e0c8e0890b87f9ac8ba: '311f765f59754bdcb4eadc04e14bd4ea' -> '272b2179c1b4476090a89880df2a8baa' move scheduled
I20240502 01:54:35.874228 19929 rebalancer_tool.cc:1074] tablet 37eda67598fe491bbb29aa62b2c4bec7: '1b50a031f2784ba498b3e52e16916205' -> '29806d3d37354c169484f6db368f8798' move scheduled
I20240502 01:54:36.216287 19929 rebalancer_tool.cc:1074] tablet 2c78951695284b1aaab411c0a041c28a: '84d3a4b174c34e02aedab1e620731359' -> '94931f4b51c9493cb3d4264ce99f5e76' move scheduled
I20240502 01:54:36.549031 19929 rebalancer_tool.cc:1074] tablet a6b5231a18a5414dbd010d4d48f08dbd: 'da3584f429104d23a1a8899b35cc2138' -> '4b7c382c9edd41babdd1a1afc904a813' move scheduled
I20240502 01:54:37.309819 19929 rebalancer_tool.cc:1074] tablet a5e03cf44eab41cc842342a138c04067: '311f765f59754bdcb4eadc04e14bd4ea' -> '29806d3d37354c169484f6db368f8798' move scheduled
I20240502 01:54:37.485607 19929 rebalancer_tool.cc:1074] tablet 62fd543d5c7c41f98926f2148ee61e7d: '1b50a031f2784ba498b3e52e16916205' -> '272b2179c1b4476090a89880df2a8baa' move scheduled
I20240502 01:54:37.641104 19929 rebalancer_tool.cc:1074] tablet 0dd8944f9fd542e29072ca501df5e112: '84d3a4b174c34e02aedab1e620731359' -> '29806d3d37354c169484f6db368f8798' move scheduled
I20240502 01:54:37.856912 19929 rebalancer_tool.cc:1074] tablet 075ded9523e44c3d92844805338643a3: 'da3584f429104d23a1a8899b35cc2138' -> '94931f4b51c9493cb3d4264ce99f5e76' move scheduled
I20240502 01:54:38.243059 19929 rebalancer_tool.cc:1074] tablet 7ce9d4281c1b46258493feb39e8d9ab2: '311f765f59754bdcb4eadc04e14bd4ea' -> '4b7c382c9edd41babdd1a1afc904a813' move scheduled
I20240502 01:54:38.655885 19929 rebalancer_tool.cc:1084] tablet 17365abbb6c1430aa278719eb8e4a648: '1b50a031f2784ba498b3e52e16916205' -> '4b7c382c9edd41babdd1a1afc904a813' move ignored: Illegal state: tablet 17365abbb6c1430aa278719eb8e4a648: replica 4b7c382c9edd41babdd1a1afc904a813 is already present
I20240502 01:54:38.656100 19929 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 01:54:41.246176 19929 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 14b25b3f4ede4b65870815b014d90449: could not find any suitable replica to move from server 84d3a4b174c34e02aedab1e620731359 to server 272b2179c1b4476090a89880df2a8baa
W20240502 01:54:41.246402 19929 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table d49bf19a92cf45ceadf02d01dbdae6ad: could not find any suitable replica to move from server 1b50a031f2784ba498b3e52e16916205 to server 84d3a4b174c34e02aedab1e620731359
W20240502 01:54:41.246728 19929 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table d49bf19a92cf45ceadf02d01dbdae6ad: could not find any suitable replica to move from server 94931f4b51c9493cb3d4264ce99f5e76 to server 29806d3d37354c169484f6db368f8798
W20240502 01:54:41.246979 19929 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table bf67e64a43cd4367a7420c6bf1793faa: could not find any suitable replica to move from server 1b50a031f2784ba498b3e52e16916205 to server 272b2179c1b4476090a89880df2a8baa
I20240502 01:54:41.817340 19929 rebalancer_tool.cc:1074] tablet 49ef898a43704b87ba077cd7b0485e9a: '1b50a031f2784ba498b3e52e16916205' -> '' move scheduled
I20240502 01:54:42.327885 19929 rebalancer_tool.cc:1074] tablet 59e507855e5043d38cef84e92adf437b: 'da3584f429104d23a1a8899b35cc2138' -> '4b7c382c9edd41babdd1a1afc904a813' move scheduled
I20240502 01:54:42.523721 19929 rebalancer_tool.cc:1084] tablet 6da59c0626c64a5988e39a831c42c268: '84d3a4b174c34e02aedab1e620731359' -> '272b2179c1b4476090a89880df2a8baa' move ignored: Illegal state: tablet 6da59c0626c64a5988e39a831c42c268: replica 272b2179c1b4476090a89880df2a8baa is already present
I20240502 01:54:42.523857 19929 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 01:54:44.073593 22100 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTabletLocations -> {remote=127.9.176.254:46743, user_credentials={real_user=slave}} blocked reactor thread for 193154us
W20240502 01:54:44.294279 22100 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTabletLocations -> {remote=127.9.176.254:46743, user_credentials={real_user=slave}} blocked reactor thread for 75766.3us
W20240502 01:54:45.342530 19929 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 14b25b3f4ede4b65870815b014d90449: could not find any suitable replica to move from server 84d3a4b174c34e02aedab1e620731359 to server 272b2179c1b4476090a89880df2a8baa
W20240502 01:54:45.342671 19929 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table d49bf19a92cf45ceadf02d01dbdae6ad: could not find any suitable replica to move from server 311f765f59754bdcb4eadc04e14bd4ea to server 272b2179c1b4476090a89880df2a8baa
W20240502 01:54:45.342756 19929 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 952912d7ec8d493bbb8985fffe9aa918: could not find any suitable replica to move from server da3584f429104d23a1a8899b35cc2138 to server 94931f4b51c9493cb3d4264ce99f5e76
I20240502 01:54:45.492367 19929 rebalancer_tool.cc:1074] tablet 3b3fa060e8d64911bec810ec07a358d2: '1b50a031f2784ba498b3e52e16916205' -> '' move scheduled
I20240502 01:54:45.818272 19929 rebalancer_tool.cc:1074] tablet 3ca5c4988ff74e3ca8d7a5614bcb93e2: '84d3a4b174c34e02aedab1e620731359' -> '272b2179c1b4476090a89880df2a8baa' move scheduled
I20240502 01:54:45.974663 19929 rebalancer_tool.cc:1084] tablet df16e178e04040699201f325ce9e2979: '84d3a4b174c34e02aedab1e620731359' -> '4b7c382c9edd41babdd1a1afc904a813' move ignored: Illegal state: tablet df16e178e04040699201f325ce9e2979: replica 4b7c382c9edd41babdd1a1afc904a813 is already present
I20240502 01:54:45.974884 19929 rebalancer_tool.cc:756] re-synchronizing cluster state
Illegal state: tablet server 29806d3d37354c169484f6db368f8798 (127.9.176.197:35011): unacceptable health status UNAVAILABLE

I20240502 01:54:49.637881 19925 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
       Statistic       |   Value
-----------------------+-----------
 Minimum Replica Count | 0
 Maximum Replica Count | 23
 Average Replica Count | 11.250000

Per-table replica distribution summary:
 Replica Skew |  Value
--------------+----------
 Minimum      | 3
 Maximum      | 3
 Average      | 3.000000


stderr: W20240502 01:54:27.590387 19930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:54:27.598022 19930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:54:27.644730 19930 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 01:54:31.528913 19930 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240502 01:54:33.396402 20664 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTableLocations -> {remote=127.9.176.254:46743, user_credentials={real_user=slave}} blocked reactor thread for 69313.2us
W20240502 01:54:34.979081 19930 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table bf67e64a43cd4367a7420c6bf1793faa: could not find any suitable replica to move from server 311f765f59754bdcb4eadc04e14bd4ea to server 4b7c382c9edd41babdd1a1afc904a813
W20240502 01:54:34.979481 19930 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 952912d7ec8d493bbb8985fffe9aa918: could not find any suitable replica to move from server 1b50a031f2784ba498b3e52e16916205 to server 29806d3d37354c169484f6db368f8798
W20240502 01:54:34.979916 19930 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table bf67e64a43cd4367a7420c6bf1793faa: could not find any suitable replica to move from server da3584f429104d23a1a8899b35cc2138 to server 94931f4b51c9493cb3d4264ce99f5e76
W20240502 01:54:34.980104 19930 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 952912d7ec8d493bbb8985fffe9aa918: could not find any suitable replica to move from server 84d3a4b174c34e02aedab1e620731359 to server 94931f4b51c9493cb3d4264ce99f5e76
W20240502 01:54:34.980273 19930 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 40a6be2c1a2744428cea56b78225e44e: could not find any suitable replica to move from server 1b50a031f2784ba498b3e52e16916205 to server 29806d3d37354c169484f6db368f8798
W20240502 01:54:34.980489 19930 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c516e7439e734f02a0947c3680cf79e2: could not find any suitable replica to move from server 84d3a4b174c34e02aedab1e620731359 to server 4b7c382c9edd41babdd1a1afc904a813
W20240502 01:54:34.980599 19930 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 73ba62f6c85943bf85dfaaf459d924f0: could not find any suitable replica to move from server 1b50a031f2784ba498b3e52e16916205 to server 94931f4b51c9493cb3d4264ce99f5e76
W20240502 01:54:34.980792 19930 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c5e1575e337a4256a90ce5b685b3ff28: could not find any suitable replica to move from server 1b50a031f2784ba498b3e52e16916205 to server 272b2179c1b4476090a89880df2a8baa
W20240502 01:54:34.980957 19930 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table d49bf19a92cf45ceadf02d01dbdae6ad: could not find any suitable replica to move from server 311f765f59754bdcb4eadc04e14bd4ea to server 4b7c382c9edd41babdd1a1afc904a813
W20240502 01:54:34.981072 19930 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 73ba62f6c85943bf85dfaaf459d924f0: could not find any suitable replica to move from server da3584f429104d23a1a8899b35cc2138 to server 29806d3d37354c169484f6db368f8798
W20240502 01:54:34.981171 19930 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 40a6be2c1a2744428cea56b78225e44e: could not find any suitable replica to move from server da3584f429104d23a1a8899b35cc2138 to server 4b7c382c9edd41babdd1a1afc904a813
W20240502 01:54:34.981281 19930 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 14b25b3f4ede4b65870815b014d90449: could not find any suitable replica to move from server 311f765f59754bdcb4eadc04e14bd4ea to server 94931f4b51c9493cb3d4264ce99f5e76
W20240502 01:54:34.981374 19930 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1bd69df0513c4c4db4dd6d6526efee4b: could not find any suitable replica to move from server 1b50a031f2784ba498b3e52e16916205 to server 272b2179c1b4476090a89880df2a8baa
W20240502 01:54:34.981479 19930 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table bf67e64a43cd4367a7420c6bf1793faa: could not find any suitable replica to move from server 1b50a031f2784ba498b3e52e16916205 to server 272b2179c1b4476090a89880df2a8baa
W20240502 01:54:34.981556 19930 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 952912d7ec8d493bbb8985fffe9aa918: could not find any suitable replica to move from server 84d3a4b174c34e02aedab1e620731359 to server 94931f4b51c9493cb3d4264ce99f5e76
W20240502 01:54:34.981705 19930 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 60aa7e7a81564c3a888cc9b68700c9ad: could not find any suitable replica to move from server 1b50a031f2784ba498b3e52e16916205 to server 4b7c382c9edd41babdd1a1afc904a813
W20240502 01:54:34.981803 19930 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c5e1575e337a4256a90ce5b685b3ff28: could not find any suitable replica to move from server 84d3a4b174c34e02aedab1e620731359 to server 29806d3d37354c169484f6db368f8798
W20240502 01:54:34.981926 19930 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1bd69df0513c4c4db4dd6d6526efee4b: could not find any suitable replica to move from server 311f765f59754bdcb4eadc04e14bd4ea to server 94931f4b51c9493cb3d4264ce99f5e76
W20240502 01:54:34.982029 19930 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table d49bf19a92cf45ceadf02d01dbdae6ad: could not find any suitable replica to move from server 1b50a031f2784ba498b3e52e16916205 to server 272b2179c1b4476090a89880df2a8baa
I20240502 01:54:35.285434 19930 rebalancer_tool.cc:1074] tablet 60ff2e50b1d74e0c8e0890b87f9ac8ba: '311f765f59754bdcb4eadc04e14bd4ea' -> '94931f4b51c9493cb3d4264ce99f5e76' move scheduled
I20240502 01:54:35.541563 19930 rebalancer_tool.cc:1074] tablet 38a60d7db47149ff90444e5276fc8935: '84d3a4b174c34e02aedab1e620731359' -> '272b2179c1b4476090a89880df2a8baa' move scheduled
I20240502 01:54:36.422993 19930 rebalancer_tool.cc:1074] tablet 17365abbb6c1430aa278719eb8e4a648: '1b50a031f2784ba498b3e52e16916205' -> '4b7c382c9edd41babdd1a1afc904a813' move scheduled
I20240502 01:54:36.810056 19930 rebalancer_tool.cc:1074] tablet 3ca5c4988ff74e3ca8d7a5614bcb93e2: 'da3584f429104d23a1a8899b35cc2138' -> '29806d3d37354c169484f6db368f8798' move scheduled
I20240502 01:54:37.040625 19930 rebalancer_tool.cc:1074] tablet 49ef898a43704b87ba077cd7b0485e9a: '311f765f59754bdcb4eadc04e14bd4ea' -> '94931f4b51c9493cb3d4264ce99f5e76' move scheduled
I20240502 01:54:37.352883 19930 rebalancer_tool.cc:1074] tablet 0dd8944f9fd542e29072ca501df5e112: '84d3a4b174c34e02aedab1e620731359' -> '272b2179c1b4476090a89880df2a8baa' move scheduled
I20240502 01:54:37.699187 19930 rebalancer_tool.cc:1084] tablet f07e9b433e554c85b3c25b7e708f8b2f: '1b50a031f2784ba498b3e52e16916205' -> '4b7c382c9edd41babdd1a1afc904a813' move ignored: Illegal state: Request specified cas_config_opid_index of 4 but the committed config has opid_index of 5
I20240502 01:54:37.699381 19930 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 01:54:40.251065 19930 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table bf67e64a43cd4367a7420c6bf1793faa: could not find any suitable replica to move from server 1b50a031f2784ba498b3e52e16916205 to server da3584f429104d23a1a8899b35cc2138
W20240502 01:54:40.251235 19930 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table d49bf19a92cf45ceadf02d01dbdae6ad: could not find any suitable replica to move from server 311f765f59754bdcb4eadc04e14bd4ea to server 84d3a4b174c34e02aedab1e620731359
W20240502 01:54:40.251372 19930 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 14b25b3f4ede4b65870815b014d90449: could not find any suitable replica to move from server 84d3a4b174c34e02aedab1e620731359 to server 272b2179c1b4476090a89880df2a8baa
W20240502 01:54:40.251494 19930 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1bd69df0513c4c4db4dd6d6526efee4b: could not find any suitable replica to move from server 1b50a031f2784ba498b3e52e16916205 to server 94931f4b51c9493cb3d4264ce99f5e76
I20240502 01:54:40.646178 19930 rebalancer_tool.cc:1074] tablet 2c78951695284b1aaab411c0a041c28a: '311f765f59754bdcb4eadc04e14bd4ea' -> '' move scheduled
I20240502 01:54:40.920207 19930 rebalancer_tool.cc:1074] tablet f07e9b433e554c85b3c25b7e708f8b2f: '1b50a031f2784ba498b3e52e16916205' -> '4b7c382c9edd41babdd1a1afc904a813' move scheduled
I20240502 01:54:41.302965 19930 rebalancer_tool.cc:1074] tablet 37eda67598fe491bbb29aa62b2c4bec7: '29806d3d37354c169484f6db368f8798' -> 'da3584f429104d23a1a8899b35cc2138' move scheduled
I20240502 01:54:41.484617 19930 rebalancer_tool.cc:1084] tablet 6da59c0626c64a5988e39a831c42c268: '84d3a4b174c34e02aedab1e620731359' -> '272b2179c1b4476090a89880df2a8baa' move ignored: Illegal state: tablet 6da59c0626c64a5988e39a831c42c268: replica 272b2179c1b4476090a89880df2a8baa is already present
I20240502 01:54:41.484820 19930 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 01:54:43.313746 22029 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTableLocations -> {remote=127.9.176.254:46743, user_credentials={real_user=slave}} blocked reactor thread for 47091.3us
W20240502 01:54:45.357066 19930 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c5e1575e337a4256a90ce5b685b3ff28: could not find any suitable replica to move from server 84d3a4b174c34e02aedab1e620731359 to server 272b2179c1b4476090a89880df2a8baa
I20240502 01:54:45.627974 19930 rebalancer_tool.cc:1074] tablet df16e178e04040699201f325ce9e2979: '84d3a4b174c34e02aedab1e620731359' -> '4b7c382c9edd41babdd1a1afc904a813' move scheduled
I20240502 01:54:45.880791 19930 rebalancer_tool.cc:1074] tablet b40f968655774ce887dff8b7f4651501: '1b50a031f2784ba498b3e52e16916205' -> '29806d3d37354c169484f6db368f8798' move scheduled
I20240502 01:54:46.003527 19930 rebalancer_tool.cc:1084] tablet a5e03cf44eab41cc842342a138c04067: 'da3584f429104d23a1a8899b35cc2138' -> '272b2179c1b4476090a89880df2a8baa' move ignored: Illegal state: tablet a5e03cf44eab41cc842342a138c04067: replica 272b2179c1b4476090a89880df2a8baa is already present
I20240502 01:54:46.004454 19930 rebalancer_tool.cc:756] re-synchronizing cluster state
Illegal state: tablet server 29806d3d37354c169484f6db368f8798 (127.9.176.197:35011): unacceptable health status UNAVAILABLE

W20240502 01:54:49.657500 17832 consensus_peers.cc:485] T 17365abbb6c1430aa278719eb8e4a648 P da3584f429104d23a1a8899b35cc2138 -> Peer 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198:36865): Couldn't send request to peer 4b7c382c9edd41babdd1a1afc904a813. Status: Network error: Client connection negotiation failed: client connection to 127.9.176.198:36865: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 01:54:49.677239 19923 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
       Statistic       |   Value
-----------------------+-----------
 Minimum Replica Count | 0
 Maximum Replica Count | 23
 Average Replica Count | 11.250000

Per-table replica distribution summary:
 Replica Skew |  Value
--------------+----------
 Minimum      | 3
 Maximum      | 3
 Average      | 3.000000


stderr: W20240502 01:54:27.015210 19926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:54:27.015700 19926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:54:27.036401 19926 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 01:54:30.022183 19926 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240502 01:54:31.818184 19926 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c5e1575e337a4256a90ce5b685b3ff28: could not find any suitable replica to move from server 84d3a4b174c34e02aedab1e620731359 to server 94931f4b51c9493cb3d4264ce99f5e76
W20240502 01:54:31.818516 19926 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 73ba62f6c85943bf85dfaaf459d924f0: could not find any suitable replica to move from server 1b50a031f2784ba498b3e52e16916205 to server 272b2179c1b4476090a89880df2a8baa
W20240502 01:54:31.818679 19926 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 40a6be2c1a2744428cea56b78225e44e: could not find any suitable replica to move from server 311f765f59754bdcb4eadc04e14bd4ea to server 94931f4b51c9493cb3d4264ce99f5e76
W20240502 01:54:31.818953 19926 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table bf67e64a43cd4367a7420c6bf1793faa: could not find any suitable replica to move from server 311f765f59754bdcb4eadc04e14bd4ea to server 4b7c382c9edd41babdd1a1afc904a813
W20240502 01:54:31.819085 19926 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table d49bf19a92cf45ceadf02d01dbdae6ad: could not find any suitable replica to move from server 311f765f59754bdcb4eadc04e14bd4ea to server 29806d3d37354c169484f6db368f8798
W20240502 01:54:31.819209 19926 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 952912d7ec8d493bbb8985fffe9aa918: could not find any suitable replica to move from server 84d3a4b174c34e02aedab1e620731359 to server 94931f4b51c9493cb3d4264ce99f5e76
W20240502 01:54:31.819432 19926 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 60aa7e7a81564c3a888cc9b68700c9ad: could not find any suitable replica to move from server 1b50a031f2784ba498b3e52e16916205 to server 272b2179c1b4476090a89880df2a8baa
W20240502 01:54:31.819667 19926 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 14b25b3f4ede4b65870815b014d90449: could not find any suitable replica to move from server 1b50a031f2784ba498b3e52e16916205 to server 29806d3d37354c169484f6db368f8798
W20240502 01:54:31.819801 19926 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1bd69df0513c4c4db4dd6d6526efee4b: could not find any suitable replica to move from server 84d3a4b174c34e02aedab1e620731359 to server 94931f4b51c9493cb3d4264ce99f5e76
W20240502 01:54:31.819934 19926 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c516e7439e734f02a0947c3680cf79e2: could not find any suitable replica to move from server 84d3a4b174c34e02aedab1e620731359 to server 272b2179c1b4476090a89880df2a8baa
W20240502 01:54:31.820050 19926 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c5e1575e337a4256a90ce5b685b3ff28: could not find any suitable replica to move from server da3584f429104d23a1a8899b35cc2138 to server 272b2179c1b4476090a89880df2a8baa
W20240502 01:54:31.820561 19926 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table bf67e64a43cd4367a7420c6bf1793faa: could not find any suitable replica to move from server 1b50a031f2784ba498b3e52e16916205 to server 4b7c382c9edd41babdd1a1afc904a813
W20240502 01:54:31.820712 19926 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table d49bf19a92cf45ceadf02d01dbdae6ad: could not find any suitable replica to move from server da3584f429104d23a1a8899b35cc2138 to server 29806d3d37354c169484f6db368f8798
W20240502 01:54:31.820868 19926 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 952912d7ec8d493bbb8985fffe9aa918: could not find any suitable replica to move from server 1b50a031f2784ba498b3e52e16916205 to server 94931f4b51c9493cb3d4264ce99f5e76
W20240502 01:54:31.820991 19926 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 73ba62f6c85943bf85dfaaf459d924f0: could not find any suitable replica to move from server da3584f429104d23a1a8899b35cc2138 to server 272b2179c1b4476090a89880df2a8baa
I20240502 01:54:32.314347 19926 rebalancer_tool.cc:1074] tablet 91e0e82526c0489a8908e250bf3c851f: 'da3584f429104d23a1a8899b35cc2138' -> '29806d3d37354c169484f6db368f8798' move scheduled
I20240502 01:54:32.592243 19926 rebalancer_tool.cc:1074] tablet 768e7cbf63a94d3a815ac77a4d0acf41: '1b50a031f2784ba498b3e52e16916205' -> '272b2179c1b4476090a89880df2a8baa' move scheduled
I20240502 01:54:32.944645 19926 rebalancer_tool.cc:1074] tablet 37eda67598fe491bbb29aa62b2c4bec7: '84d3a4b174c34e02aedab1e620731359' -> '4b7c382c9edd41babdd1a1afc904a813' move scheduled
I20240502 01:54:33.155225 19926 rebalancer_tool.cc:1074] tablet 6da59c0626c64a5988e39a831c42c268: '311f765f59754bdcb4eadc04e14bd4ea' -> '94931f4b51c9493cb3d4264ce99f5e76' move scheduled
I20240502 01:54:33.487247 19926 rebalancer_tool.cc:1074] tablet 2045e08a200148c69326351f38fd484d: 'da3584f429104d23a1a8899b35cc2138' -> '29806d3d37354c169484f6db368f8798' move scheduled
I20240502 01:54:34.370687 19926 rebalancer_tool.cc:1074] tablet 2c78951695284b1aaab411c0a041c28a: '1b50a031f2784ba498b3e52e16916205' -> '272b2179c1b4476090a89880df2a8baa' move scheduled
I20240502 01:54:34.719971 19926 rebalancer_tool.cc:1074] tablet b40f968655774ce887dff8b7f4651501: '84d3a4b174c34e02aedab1e620731359' -> '4b7c382c9edd41babdd1a1afc904a813' move scheduled
I20240502 01:54:35.336181 19926 rebalancer_tool.cc:1084] tablet 60ff2e50b1d74e0c8e0890b87f9ac8ba: '311f765f59754bdcb4eadc04e14bd4ea' -> '94931f4b51c9493cb3d4264ce99f5e76' move ignored: Illegal state: tablet 60ff2e50b1d74e0c8e0890b87f9ac8ba: replica 94931f4b51c9493cb3d4264ce99f5e76 is already present
I20240502 01:54:35.336361 19926 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 01:54:39.569543 19926 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 952912d7ec8d493bbb8985fffe9aa918: could not find any suitable replica to move from server 84d3a4b174c34e02aedab1e620731359 to server da3584f429104d23a1a8899b35cc2138
W20240502 01:54:39.569965 19926 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c516e7439e734f02a0947c3680cf79e2: could not find any suitable replica to move from server 1b50a031f2784ba498b3e52e16916205 to server 4b7c382c9edd41babdd1a1afc904a813
W20240502 01:54:39.570190 19926 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table d49bf19a92cf45ceadf02d01dbdae6ad: could not find any suitable replica to move from server 311f765f59754bdcb4eadc04e14bd4ea to server 29806d3d37354c169484f6db368f8798
W20240502 01:54:39.570477 19926 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1bd69df0513c4c4db4dd6d6526efee4b: could not find any suitable replica to move from server da3584f429104d23a1a8899b35cc2138 to server 29806d3d37354c169484f6db368f8798
W20240502 01:54:39.570623 19926 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table bf67e64a43cd4367a7420c6bf1793faa: could not find any suitable replica to move from server 311f765f59754bdcb4eadc04e14bd4ea to server 94931f4b51c9493cb3d4264ce99f5e76
W20240502 01:54:39.570793 19926 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 73ba62f6c85943bf85dfaaf459d924f0: could not find any suitable replica to move from server 1b50a031f2784ba498b3e52e16916205 to server 272b2179c1b4476090a89880df2a8baa
I20240502 01:54:39.922474 19926 rebalancer_tool.cc:1074] tablet c91f74f46b04425aa5393515cb9f781c: '311f765f59754bdcb4eadc04e14bd4ea' -> '4b7c382c9edd41babdd1a1afc904a813' move scheduled
I20240502 01:54:40.231006 19926 rebalancer_tool.cc:1074] tablet 05b67d9912f047bfb6a2bed90b82ae15: '84d3a4b174c34e02aedab1e620731359' -> '94931f4b51c9493cb3d4264ce99f5e76' move scheduled
I20240502 01:54:40.922618 19926 rebalancer_tool.cc:1074] tablet 3b3fa060e8d64911bec810ec07a358d2: '1b50a031f2784ba498b3e52e16916205' -> 'da3584f429104d23a1a8899b35cc2138' move scheduled
I20240502 01:54:41.085350 19926 rebalancer_tool.cc:1084] tablet 17365abbb6c1430aa278719eb8e4a648: '1b50a031f2784ba498b3e52e16916205' -> '29806d3d37354c169484f6db368f8798' move ignored: Not found: Peer with uuid 1b50a031f2784ba498b3e52e16916205 not found in consensus config
I20240502 01:54:41.085507 19926 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 01:54:41.922000 21894 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTableLocations -> {remote=127.9.176.254:46743, user_credentials={real_user=slave}} blocked reactor thread for 44120.6us
W20240502 01:54:43.103053 19926 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 40a6be2c1a2744428cea56b78225e44e: could not find any suitable replica to move from server 311f765f59754bdcb4eadc04e14bd4ea to server 29806d3d37354c169484f6db368f8798
I20240502 01:54:43.346395 19926 rebalancer_tool.cc:1074] tablet a6b5231a18a5414dbd010d4d48f08dbd: '1b50a031f2784ba498b3e52e16916205' -> '29806d3d37354c169484f6db368f8798' move scheduled
I20240502 01:54:43.589632 19926 rebalancer_tool.cc:1074] tablet 88a137bcfc76410583f910b377d1be7f: '311f765f59754bdcb4eadc04e14bd4ea' -> '272b2179c1b4476090a89880df2a8baa' move scheduled
I20240502 01:54:44.063089 19926 rebalancer_tool.cc:1074] tablet 62fd543d5c7c41f98926f2148ee61e7d: '84d3a4b174c34e02aedab1e620731359' -> '94931f4b51c9493cb3d4264ce99f5e76' move scheduled
I20240502 01:54:44.663611 19926 rebalancer_tool.cc:1074] tablet fadcc8fe6ae047c1b142514aad8bebba: '84d3a4b174c34e02aedab1e620731359' -> '4b7c382c9edd41babdd1a1afc904a813' move scheduled
I20240502 01:54:44.987557 19926 rebalancer_tool.cc:1074] tablet 38a60d7db47149ff90444e5276fc8935: '1b50a031f2784ba498b3e52e16916205' -> '29806d3d37354c169484f6db368f8798' move scheduled
I20240502 01:54:45.900101 19926 rebalancer_tool.cc:1136] tablet b40f968655774ce887dff8b7f4651501: '84d3a4b174c34e02aedab1e620731359' -> '4b7c382c9edd41babdd1a1afc904a813' move completed: OK
I20240502 01:54:46.000902 19926 rebalancer_tool.cc:1136] tablet 768e7cbf63a94d3a815ac77a4d0acf41: '1b50a031f2784ba498b3e52e16916205' -> '272b2179c1b4476090a89880df2a8baa' move completed: OK
I20240502 01:54:46.159268 19926 rebalancer_tool.cc:1136] tablet 37eda67598fe491bbb29aa62b2c4bec7: '84d3a4b174c34e02aedab1e620731359' -> '4b7c382c9edd41babdd1a1afc904a813' move completed: Incomplete: tablet 37eda67598fe491bbb29aa62b2c4bec7, TS 84d3a4b174c34e02aedab1e620731359 -> TS 4b7c382c9edd41babdd1a1afc904a813 move failed, target replica disappeared
I20240502 01:54:46.231017 19926 rebalancer_tool.cc:1136] tablet 91e0e82526c0489a8908e250bf3c851f: 'da3584f429104d23a1a8899b35cc2138' -> '29806d3d37354c169484f6db368f8798' move completed: OK
I20240502 01:54:46.292348 19926 rebalancer_tool.cc:1136] tablet 05b67d9912f047bfb6a2bed90b82ae15: '84d3a4b174c34e02aedab1e620731359' -> '94931f4b51c9493cb3d4264ce99f5e76' move completed: OK
I20240502 01:54:46.846704 19926 rebalancer_tool.cc:1136] tablet 6da59c0626c64a5988e39a831c42c268: '311f765f59754bdcb4eadc04e14bd4ea' -> '94931f4b51c9493cb3d4264ce99f5e76' move completed: OK
I20240502 01:54:47.126974 19926 rebalancer_tool.cc:1136] tablet c91f74f46b04425aa5393515cb9f781c: '311f765f59754bdcb4eadc04e14bd4ea' -> '4b7c382c9edd41babdd1a1afc904a813' move completed: OK
I20240502 01:54:47.838244 19926 rebalancer_tool.cc:1136] tablet 2045e08a200148c69326351f38fd484d: 'da3584f429104d23a1a8899b35cc2138' -> '29806d3d37354c169484f6db368f8798' move completed: Incomplete: tablet 2045e08a200148c69326351f38fd484d, TS da3584f429104d23a1a8899b35cc2138 -> TS 29806d3d37354c169484f6db368f8798 move failed, target replica disappeared
I20240502 01:54:48.473788 19926 rebalancer_tool.cc:1084] tablet a5e03cf44eab41cc842342a138c04067: '84d3a4b174c34e02aedab1e620731359' -> '272b2179c1b4476090a89880df2a8baa' move ignored: Illegal state: tablet a5e03cf44eab41cc842342a138c04067: replica 272b2179c1b4476090a89880df2a8baa is already present
I20240502 01:54:48.473950 19926 rebalancer_tool.cc:756] re-synchronizing cluster state
Illegal state: tablet server 29806d3d37354c169484f6db368f8798 (127.9.176.197:35011): unacceptable health status UNAVAILABLE

E20240502 01:54:49.694041  9923 external_mini_cluster.cc:943] Process with UUID 29806d3d37354c169484f6db368f8798 has crashed
E20240502 01:54:49.694198  9923 external_mini_cluster.cc:943] Process with UUID 4b7c382c9edd41babdd1a1afc904a813 has crashed
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/mini-cluster/external_mini_cluster.cc:947: Failure
Expected equality of these values:
  0
  num_crashes
    Which is: 2
At least one process crashed
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tools/rebalancer_tool-test.cc:1284: Failure
Expected: cluster_->AssertNoCrashes() doesn't generate new fatal failures in the current thread.
  Actual: it does.
I20240502 01:54:49.695906  9923 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu pid 17416
I20240502 01:54:49.703599 17538 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 01:54:49.762305 17832 consensus_peers.cc:485] T 17365abbb6c1430aa278719eb8e4a648 P da3584f429104d23a1a8899b35cc2138 -> Peer 29806d3d37354c169484f6db368f8798 (127.9.176.197:35011): Couldn't send request to peer 29806d3d37354c169484f6db368f8798. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.9.176.197:35011: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:54:49.797940 17565 consensus_peers.cc:462] T b40f968655774ce887dff8b7f4651501 P 1b50a031f2784ba498b3e52e16916205 -> Peer 29806d3d37354c169484f6db368f8798 (127.9.176.197:35011): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.9.176.197:35011: connect: Connection refused (error 111)
I20240502 01:54:49.983006  9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu with pid 17416
W20240502 01:54:50.018100 17832 consensus_peers.cc:485] T 3b3fa060e8d64911bec810ec07a358d2 P da3584f429104d23a1a8899b35cc2138 -> Peer 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198:36865): Couldn't send request to peer 4b7c382c9edd41babdd1a1afc904a813. Status: Network error: Client connection negotiation failed: client connection to 127.9.176.198:36865: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:54:50.029814 17568 connection.cc:535] client connection to 127.9.176.193:35397 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:54:50.030126 17703 connection.cc:535] client connection to 127.9.176.193:35397 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 01:54:50.030597  9923 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu pid 17549
W20240502 01:54:50.031467 17703 consensus_peers.cc:485] T d022e878d5d943eb849e53d15029aa77 P 84d3a4b174c34e02aedab1e620731359 -> Peer 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193:35397): Couldn't send request to peer 311f765f59754bdcb4eadc04e14bd4ea. Status: Network error: Client connection negotiation failed: client connection to 127.9.176.193:35397: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:50.031466 17568 consensus_peers.cc:485] T 38a60d7db47149ff90444e5276fc8935 P 1b50a031f2784ba498b3e52e16916205 -> Peer 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193:35397): Couldn't send request to peer 311f765f59754bdcb4eadc04e14bd4ea. Status: Network error: Client connection negotiation failed: client connection to 127.9.176.193:35397: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:50.043258 17674 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 01:54:50.048134 17703 consensus_peers.cc:485] T 3ca5c4988ff74e3ca8d7a5614bcb93e2 P 84d3a4b174c34e02aedab1e620731359 -> Peer 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193:35397): Couldn't send request to peer 311f765f59754bdcb4eadc04e14bd4ea. Status: Network error: Client connection negotiation failed: client connection to 127.9.176.193:35397: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:50.097956 17832 consensus_peers.cc:485] T 05b67d9912f047bfb6a2bed90b82ae15 P da3584f429104d23a1a8899b35cc2138 -> Peer 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198:36865): Couldn't send request to peer 4b7c382c9edd41babdd1a1afc904a813. Status: Network error: Client connection negotiation failed: client connection to 127.9.176.198:36865: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:54:50.193208 17700 consensus_peers.cc:485] T c91f74f46b04425aa5393515cb9f781c P 84d3a4b174c34e02aedab1e620731359 -> Peer 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198:36865): Couldn't send request to peer 4b7c382c9edd41babdd1a1afc904a813. Status: Network error: Client connection negotiation failed: client connection to 127.9.176.198:36865: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 01:54:50.331667  9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu with pid 17549
W20240502 01:54:50.385378 19806 connection.cc:535] server connection from 127.9.176.194:49149 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 01:54:50.386035  9923 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu pid 17684
I20240502 01:54:50.393234 17806 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 01:54:50.476048 17835 consensus_peers.cc:485] T 3bdd49939e994240a6d15dbd11559ff4 P da3584f429104d23a1a8899b35cc2138 -> Peer 1b50a031f2784ba498b3e52e16916205 (127.9.176.194:45617): Couldn't send request to peer 1b50a031f2784ba498b3e52e16916205. Status: Network error: Client connection negotiation failed: client connection to 127.9.176.194:45617: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:50.670783  9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu with pid 17684
W20240502 01:54:50.715981 19669 connection.cc:535] server connection from 127.9.176.195:60725 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:54:50.716632 17342 connection.cc:535] server connection from 127.9.176.195:60083 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 01:54:50.716955  9923 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu pid 17817
I20240502 01:54:50.723198 17941 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:54:50.798244 22948 raft_consensus.cc:482] T 075ded9523e44c3d92844805338643a3 P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 311f765f59754bdcb4eadc04e14bd4ea)
I20240502 01:54:50.798570 22948 raft_consensus.cc:504] T 075ded9523e44c3d92844805338643a3 P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } }
I20240502 01:54:50.800264 22948 leader_election.cc:288] T 075ded9523e44c3d92844805338643a3 P 94931f4b51c9493cb3d4264ce99f5e76 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1b50a031f2784ba498b3e52e16916205 (127.9.176.194:45617), 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193:35397)
W20240502 01:54:50.801738 19806 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.176.194:45617: connect: Connection refused (error 111)
W20240502 01:54:50.803315 19806 leader_election.cc:334] T 075ded9523e44c3d92844805338643a3 P 94931f4b51c9493cb3d4264ce99f5e76 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1b50a031f2784ba498b3e52e16916205 (127.9.176.194:45617): Network error: Client connection negotiation failed: client connection to 127.9.176.194:45617: connect: Connection refused (error 111)
W20240502 01:54:50.803586 19806 leader_election.cc:334] T 075ded9523e44c3d92844805338643a3 P 94931f4b51c9493cb3d4264ce99f5e76 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193:35397): Network error: Client connection negotiation failed: client connection to 127.9.176.193:35397: connect: Connection refused (error 111)
I20240502 01:54:50.803726 19806 leader_election.cc:302] T 075ded9523e44c3d92844805338643a3 P 94931f4b51c9493cb3d4264ce99f5e76 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94931f4b51c9493cb3d4264ce99f5e76; no voters: 1b50a031f2784ba498b3e52e16916205, 311f765f59754bdcb4eadc04e14bd4ea
I20240502 01:54:50.804083 22948 raft_consensus.cc:2726] T 075ded9523e44c3d92844805338643a3 P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:54:50.982318 22953 raft_consensus.cc:482] T fadcc8fe6ae047c1b142514aad8bebba P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 311f765f59754bdcb4eadc04e14bd4ea)
I20240502 01:54:50.982636 22953 raft_consensus.cc:504] T fadcc8fe6ae047c1b142514aad8bebba P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "4b7c382c9edd41babdd1a1afc904a813" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 36865 } attrs { promote: false } }
I20240502 01:54:50.983896 22953 leader_election.cc:288] T fadcc8fe6ae047c1b142514aad8bebba P da3584f429104d23a1a8899b35cc2138 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193:35397), 4b7c382c9edd41babdd1a1afc904a813 (127.9.176.198:36865)
I20240502 01:54:50.984131 22953 raft_consensus.cc:482] T 37eda67598fe491bbb29aa62b2c4bec7 P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 311f765f59754bdcb4eadc04e14bd4ea)
I20240502 01:54:50.984342 22953 raft_consensus.cc:504] T 37eda67598fe491bbb29aa62b2c4bec7 P da3584f429104d23a1a8899b35cc2138 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { promote: false } }
I20240502 01:54:50.985208 22953 leader_election.cc:288] T 37eda67598fe491bbb29aa62b2c4bec7 P da3584f429104d23a1a8899b35cc2138 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193:35397), 94931f4b51c9493cb3d4264ce99f5e76 (127.9.176.200:37831)
I20240502 01:54:50.985893 19872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37eda67598fe491bbb29aa62b2c4bec7" candidate_uuid: "da3584f429104d23a1a8899b35cc2138" candidate_term: 2 candidate_status { last_received { term: 1 index: 14 } } ignore_live_leader: false dest_uuid: "94931f4b51c9493cb3d4264ce99f5e76" is_pre_election: true
I20240502 01:54:50.986822  9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu with pid 17817
I20240502 01:54:50.999555 22948 raft_consensus.cc:482] T 37eda67598fe491bbb29aa62b2c4bec7 P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 311f765f59754bdcb4eadc04e14bd4ea)
I20240502 01:54:50.999826 22948 raft_consensus.cc:504] T 37eda67598fe491bbb29aa62b2c4bec7 P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } attrs { promote: false } }
I20240502 01:54:51.000768 22948 leader_election.cc:288] T 37eda67598fe491bbb29aa62b2c4bec7 P 94931f4b51c9493cb3d4264ce99f5e76 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193:35397), da3584f429104d23a1a8899b35cc2138 (127.9.176.196:46381)
W20240502 01:54:51.002225 19806 leader_election.cc:334] T 37eda67598fe491bbb29aa62b2c4bec7 P 94931f4b51c9493cb3d4264ce99f5e76 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193:35397): Network error: Client connection negotiation failed: client connection to 127.9.176.193:35397: connect: Connection refused (error 111)
W20240502 01:54:51.025339 19803 connection.cc:535] client connection to 127.9.176.196:46381 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 01:54:51.026031  9923 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu pid 19654
W20240502 01:54:51.026542 19803 leader_election.cc:334] T 37eda67598fe491bbb29aa62b2c4bec7 P 94931f4b51c9493cb3d4264ce99f5e76 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer da3584f429104d23a1a8899b35cc2138 (127.9.176.196:46381): Network error: Client connection negotiation failed: client connection to 127.9.176.196:46381: connect: Connection refused (error 111)
I20240502 01:54:51.026712 19803 leader_election.cc:302] T 37eda67598fe491bbb29aa62b2c4bec7 P 94931f4b51c9493cb3d4264ce99f5e76 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94931f4b51c9493cb3d4264ce99f5e76; no voters: 311f765f59754bdcb4eadc04e14bd4ea, da3584f429104d23a1a8899b35cc2138
I20240502 01:54:51.027179 22948 raft_consensus.cc:2726] T 37eda67598fe491bbb29aa62b2c4bec7 P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:54:51.030521 22948 raft_consensus.cc:482] T 49ef898a43704b87ba077cd7b0485e9a P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 311f765f59754bdcb4eadc04e14bd4ea)
I20240502 01:54:51.030755 22948 raft_consensus.cc:504] T 49ef898a43704b87ba077cd7b0485e9a P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { replace: true } } peers { permanent_uuid: "da3584f429104d23a1a8899b35cc2138" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 46381 } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } }
I20240502 01:54:51.031836 22948 leader_election.cc:288] T 49ef898a43704b87ba077cd7b0485e9a P 94931f4b51c9493cb3d4264ce99f5e76 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193:35397), da3584f429104d23a1a8899b35cc2138 (127.9.176.196:46381), 29806d3d37354c169484f6db368f8798 (127.9.176.197:35011)
I20240502 01:54:51.033556 19778 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 01:54:51.033947 19803 leader_election.cc:334] T 49ef898a43704b87ba077cd7b0485e9a P 94931f4b51c9493cb3d4264ce99f5e76 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer da3584f429104d23a1a8899b35cc2138 (127.9.176.196:46381): Network error: Client connection negotiation failed: client connection to 127.9.176.196:46381: connect: Connection refused (error 111)
W20240502 01:54:51.034221 19806 leader_election.cc:334] T 49ef898a43704b87ba077cd7b0485e9a P 94931f4b51c9493cb3d4264ce99f5e76 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193:35397): Network error: Client connection negotiation failed: client connection to 127.9.176.193:35397: connect: Connection refused (error 111)
I20240502 01:54:51.034337 19806 leader_election.cc:302] T 49ef898a43704b87ba077cd7b0485e9a P 94931f4b51c9493cb3d4264ce99f5e76 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 94931f4b51c9493cb3d4264ce99f5e76; no voters: 311f765f59754bdcb4eadc04e14bd4ea, da3584f429104d23a1a8899b35cc2138
I20240502 01:54:51.034652 22948 raft_consensus.cc:2726] T 49ef898a43704b87ba077cd7b0485e9a P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 01:54:51.034711 19803 leader_election.cc:334] T 49ef898a43704b87ba077cd7b0485e9a P 94931f4b51c9493cb3d4264ce99f5e76 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 29806d3d37354c169484f6db368f8798 (127.9.176.197:35011): Network error: Client connection negotiation failed: client connection to 127.9.176.197:35011: connect: Connection refused (error 111)
I20240502 01:54:51.125571 22948 raft_consensus.cc:482] T 62fd543d5c7c41f98926f2148ee61e7d P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 311f765f59754bdcb4eadc04e14bd4ea)
I20240502 01:54:51.125824 22948 raft_consensus.cc:504] T 62fd543d5c7c41f98926f2148ee61e7d P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } } peers { permanent_uuid: "272b2179c1b4476090a89880df2a8baa" member_type: VOTER last_known_addr { host: "127.9.176.199" port: 42269 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } }
I20240502 01:54:51.126643 22948 leader_election.cc:288] T 62fd543d5c7c41f98926f2148ee61e7d P 94931f4b51c9493cb3d4264ce99f5e76 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193:35397), 272b2179c1b4476090a89880df2a8baa (127.9.176.199:42269)
W20240502 01:54:51.128096 19806 leader_election.cc:334] T 62fd543d5c7c41f98926f2148ee61e7d P 94931f4b51c9493cb3d4264ce99f5e76 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 311f765f59754bdcb4eadc04e14bd4ea (127.9.176.193:35397): Network error: Client connection negotiation failed: client connection to 127.9.176.193:35397: connect: Connection refused (error 111)
I20240502 01:54:51.230603 22963 raft_consensus.cc:482] T 62fd543d5c7c41f98926f2148ee61e7d P 272b2179c1b4476090a89880df2a8baa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 311f765f59754bdcb4eadc04e14bd4ea)
I20240502 01:54:51.230834  9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu with pid 19654
W20240502 01:54:51.265826 22949 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 01:54:51.126912 (+     0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.199:42269 (local address 127.9.176.200:56033)
0502 01:54:51.127148 (+   236us) negotiation.cc:107] Waiting for socket to connect
0502 01:54:51.127165 (+    17us) client_negotiation.cc:174] Beginning negotiation
0502 01:54:51.127289 (+   124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 01:54:51.264991 (+137702us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.199:42269: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":161}
I20240502 01:54:51.266512  9923 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu pid 19787
W20240502 01:54:51.267153 19803 leader_election.cc:334] T 62fd543d5c7c41f98926f2148ee61e7d P 94931f4b51c9493cb3d4264ce99f5e76 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 272b2179c1b4476090a89880df2a8baa (127.9.176.199:42269): Network error: Client connection negotiation failed: client connection to 127.9.176.199:42269: connect: Connection refused (error 111)
I20240502 01:54:51.267383 19803 leader_election.cc:302] T 62fd543d5c7c41f98926f2148ee61e7d P 94931f4b51c9493cb3d4264ce99f5e76 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94931f4b51c9493cb3d4264ce99f5e76; no voters: 272b2179c1b4476090a89880df2a8baa, 311f765f59754bdcb4eadc04e14bd4ea
I20240502 01:54:51.267861 22948 raft_consensus.cc:2726] T 62fd543d5c7c41f98926f2148ee61e7d P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:54:51.273329 19908 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:54:51.473692 22948 raft_consensus.cc:482] T 91e0e82526c0489a8908e250bf3c851f P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1b50a031f2784ba498b3e52e16916205)
I20240502 01:54:51.473997 22948 raft_consensus.cc:504] T 91e0e82526c0489a8908e250bf3c851f P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1b50a031f2784ba498b3e52e16916205" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 45617 } attrs { replace: true } } peers { permanent_uuid: "29806d3d37354c169484f6db368f8798" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "94931f4b51c9493cb3d4264ce99f5e76" member_type: VOTER last_known_addr { host: "127.9.176.200" port: 37831 } attrs { promote: false } } peers { permanent_uuid: "311f765f59754bdcb4eadc04e14bd4ea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35397 } attrs { promote: false } }
I20240502 01:54:51.474293 22967 raft_consensus.cc:482] T 2c78951695284b1aaab411c0a041c28a P 94931f4b51c9493cb3d4264ce99f5e76 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1b50a031f2784ba498b3e52e16916205)
I20240502 01:54:51.474539  9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu with pid 19787
I20240502 01:54:51.505049  9923 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu pid 17325
I20240502 01:54:51.506142 17385 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:54:51.767287  9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu with pid 17325
2024-05-02T01:54:51Z chronyd exiting
I20240502 01:54:51.831691  9923 test_util.cc:175] -----------------------------------------------
I20240502 01:54:51.831804  9923 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714614751024524-9923-0
[  FAILED  ] ConcurrentRebalancersTest.TwoConcurrentRebalancers/1, where GetParam() = 4-byte object <01-00 00-00> (52303 ms)
[----------] 1 test from ConcurrentRebalancersTest (52303 ms total)

[----------] 1 test from RebalancerAndSingleReplicaTablets
[ RUN      ] RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove/1
I20240502 01:54:51.834074  9923 test_util.cc:268] Using random seed: -477238595
I20240502 01:54:51.836050  9923 ts_itest-base.cc:112] Starting cluster with:
I20240502 01:54:51.836122  9923 ts_itest-base.cc:113] --------------
I20240502 01:54:51.836167  9923 ts_itest-base.cc:114] 4 tablet servers
I20240502 01:54:51.836227  9923 ts_itest-base.cc:115] 1 replicas per TS
I20240502 01:54:51.836285  9923 ts_itest-base.cc:116] --------------
2024-05-02T01:54:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T01:54:51Z Disabled control of system clock
I20240502 01:54:51.891822  9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
/tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:44139
--webserver_interface=127.9.176.254
--webserver_port=0
--builtin_ntp_servers=127.9.176.212:43017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.176.254:44139
--raft_prepare_replacement_before_eviction=true
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false with env {}
W20240502 01:54:52.266873 22976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:54:52.267237 22976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:54:52.267364 22976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:54:52.275476 22976 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:54:52.275661 22976 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 01:54:52.275749 22976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:54:52.275835 22976 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 01:54:52.275903 22976 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 01:54:52.285786 22976 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:43017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.9.176.254:44139
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:44139
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.9.176.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true

Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:54:52.287225 22976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:54:52.289230 22976 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:54:52.301673 22982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:52.302067 22981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:52.302460 22984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:54:52.303205 22976 server_base.cc:1034] running on GCE node
I20240502 01:54:52.304100 22976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:54:52.305755 22976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:54:52.306989 22976 hybrid_clock.cc:648] HybridClock initialized: now 1714614892306946 us; error 39 us; skew 500 ppm
I20240502 01:54:52.307395 22976 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:54:52.310026 22976 webserver.cc:460] Webserver started at http://127.9.176.254:42177/ using document root <none> and password file <none>
I20240502 01:54:52.310854 22976 fs_manager.cc:359] Metadata directory not provided
I20240502 01:54:52.310983 22976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:54:52.311340 22976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:54:52.314239 22976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "4378e3dea314412ab6ba0f0dc7a84669"
format_stamp: "Formatted at 2024-05-02 01:54:52 on dist-test-slave-j3hh"
I20240502 01:54:52.315109 22976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "4378e3dea314412ab6ba0f0dc7a84669"
format_stamp: "Formatted at 2024-05-02 01:54:52 on dist-test-slave-j3hh"
I20240502 01:54:52.320896 22976 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.006s	sys 0.002s
I20240502 01:54:52.324802 22990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:54:52.326268 22976 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.002s
I20240502 01:54:52.326483 22976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/wal
uuid: "4378e3dea314412ab6ba0f0dc7a84669"
format_stamp: "Formatted at 2024-05-02 01:54:52 on dist-test-slave-j3hh"
I20240502 01:54:52.326718 22976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:54:52.356891 22976 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:54:52.357909 22976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:54:52.358232 22976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:54:52.379629 22976 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.254:44139
I20240502 01:54:52.379632 23041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.254:44139 every 8 connection(s)
I20240502 01:54:52.381311 22976 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 01:54:52.385818 23042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:52.389067  9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu as pid 22976
I20240502 01:54:52.389315  9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 01:54:52.400328 23042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4378e3dea314412ab6ba0f0dc7a84669: Bootstrap starting.
I20240502 01:54:52.404434 23042 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4378e3dea314412ab6ba0f0dc7a84669: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:52.405593 23042 log.cc:824] T 00000000000000000000000000000000 P 4378e3dea314412ab6ba0f0dc7a84669: Log is configured to *not* fsync() on all Append() calls
I20240502 01:54:52.408347 23042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4378e3dea314412ab6ba0f0dc7a84669: No bootstrap required, opened a new log
I20240502 01:54:52.414822 23042 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4378e3dea314412ab6ba0f0dc7a84669 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4378e3dea314412ab6ba0f0dc7a84669" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 44139 } }
I20240502 01:54:52.415247 23042 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4378e3dea314412ab6ba0f0dc7a84669 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:52.415364 23042 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4378e3dea314412ab6ba0f0dc7a84669 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4378e3dea314412ab6ba0f0dc7a84669, State: Initialized, Role: FOLLOWER
I20240502 01:54:52.416090 23042 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4378e3dea314412ab6ba0f0dc7a84669 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4378e3dea314412ab6ba0f0dc7a84669" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 44139 } }
I20240502 01:54:52.416324 23042 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4378e3dea314412ab6ba0f0dc7a84669 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:54:52.416460 23042 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4378e3dea314412ab6ba0f0dc7a84669 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:54:52.416613 23042 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4378e3dea314412ab6ba0f0dc7a84669 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:52.418802 23042 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4378e3dea314412ab6ba0f0dc7a84669 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4378e3dea314412ab6ba0f0dc7a84669" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 44139 } }
I20240502 01:54:52.419329 23042 leader_election.cc:302] T 00000000000000000000000000000000 P 4378e3dea314412ab6ba0f0dc7a84669 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4378e3dea314412ab6ba0f0dc7a84669; no voters: 
I20240502 01:54:52.420001 23042 leader_election.cc:288] T 00000000000000000000000000000000 P 4378e3dea314412ab6ba0f0dc7a84669 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240502 01:54:52.420146 23047 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4378e3dea314412ab6ba0f0dc7a84669 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:52.420853 23047 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4378e3dea314412ab6ba0f0dc7a84669 [term 1 LEADER]: Becoming Leader. State: Replica: 4378e3dea314412ab6ba0f0dc7a84669, State: Running, Role: LEADER
I20240502 01:54:52.421418 23047 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4378e3dea314412ab6ba0f0dc7a84669 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4378e3dea314412ab6ba0f0dc7a84669" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 44139 } }
I20240502 01:54:52.422149 23042 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4378e3dea314412ab6ba0f0dc7a84669 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 01:54:52.424410 23049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4378e3dea314412ab6ba0f0dc7a84669 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4378e3dea314412ab6ba0f0dc7a84669. Latest consensus state: current_term: 1 leader_uuid: "4378e3dea314412ab6ba0f0dc7a84669" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4378e3dea314412ab6ba0f0dc7a84669" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 44139 } } }
I20240502 01:54:52.424860 23049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4378e3dea314412ab6ba0f0dc7a84669 [sys.catalog]: This master's current role is: LEADER
I20240502 01:54:52.424850 23048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4378e3dea314412ab6ba0f0dc7a84669 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4378e3dea314412ab6ba0f0dc7a84669" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4378e3dea314412ab6ba0f0dc7a84669" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 44139 } } }
I20240502 01:54:52.425185 23048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4378e3dea314412ab6ba0f0dc7a84669 [sys.catalog]: This master's current role is: LEADER
I20240502 01:54:52.427309 23054 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 01:54:52.433158 23054 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 01:54:52.443297 23054 catalog_manager.cc:1347] Generated new cluster ID: c2ebfc429ebc4c07abc514a69053489d
I20240502 01:54:52.443434 23054 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 01:54:52.474419 23054 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 01:54:52.475847 23054 catalog_manager.cc:1504] Loading token signing keys...
I20240502 01:54:52.486670 23054 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 4378e3dea314412ab6ba0f0dc7a84669: Generated new TSK 0
I20240502 01:54:52.487687 23054 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 01:54:52.498387  9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
/tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.193:0
--local_ip_for_outbound_sockets=127.9.176.193
--webserver_interface=127.9.176.193
--webserver_port=0
--tserver_master_addrs=127.9.176.254:44139
--builtin_ntp_servers=127.9.176.212:43017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 01:54:52.897313 23066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:54:52.897691 23066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:54:52.897794 23066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:54:52.905889 23066 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:54:52.906026 23066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:54:52.906155 23066 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.193
I20240502 01:54:52.915854 23066 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:43017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.9.176.193
--webserver_port=0
--tserver_master_addrs=127.9.176.254:44139
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.193
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:54:52.917243 23066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:54:52.918959 23066 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:54:52.931634 23071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:52.931988 23072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:52.932718 23074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:54:52.933503 23066 server_base.cc:1034] running on GCE node
I20240502 01:54:52.934193 23066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:54:52.935276 23066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:54:52.936511 23066 hybrid_clock.cc:648] HybridClock initialized: now 1714614892936456 us; error 41 us; skew 500 ppm
I20240502 01:54:52.936817 23066 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:54:52.939226 23066 webserver.cc:460] Webserver started at http://127.9.176.193:44007/ using document root <none> and password file <none>
I20240502 01:54:52.940109 23066 fs_manager.cc:359] Metadata directory not provided
I20240502 01:54:52.940219 23066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:54:52.940518 23066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:54:52.943320 23066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "29f84c922f9f47f8bd7c360af478d359"
format_stamp: "Formatted at 2024-05-02 01:54:52 on dist-test-slave-j3hh"
I20240502 01:54:52.944263 23066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "29f84c922f9f47f8bd7c360af478d359"
format_stamp: "Formatted at 2024-05-02 01:54:52 on dist-test-slave-j3hh"
I20240502 01:54:52.950263 23066 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.005s	sys 0.000s
I20240502 01:54:52.954107 23080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:54:52.955650 23066 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.002s
I20240502 01:54:52.955852 23066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "29f84c922f9f47f8bd7c360af478d359"
format_stamp: "Formatted at 2024-05-02 01:54:52 on dist-test-slave-j3hh"
I20240502 01:54:52.956063 23066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:54:52.998090 23066 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:54:52.999028 23066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:54:52.999301 23066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:54:53.000334 23066 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:54:53.002004 23066 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:54:53.002104 23066 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:54:53.002194 23066 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:54:53.002251 23066 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:54:53.036530 23066 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.193:42839
I20240502 01:54:53.036613 23192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.193:42839 every 8 connection(s)
I20240502 01:54:53.037875 23066 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 01:54:53.042050  9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu as pid 23066
I20240502 01:54:53.042280  9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 01:54:53.046041  9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
/tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.194:0
--local_ip_for_outbound_sockets=127.9.176.194
--webserver_interface=127.9.176.194
--webserver_port=0
--tserver_master_addrs=127.9.176.254:44139
--builtin_ntp_servers=127.9.176.212:43017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:54:53.052237 23193 heartbeater.cc:347] Connected to a master server at 127.9.176.254:44139
I20240502 01:54:53.052589 23193 heartbeater.cc:464] Registering TS with master...
I20240502 01:54:53.053531 23193 heartbeater.cc:511] Master 127.9.176.254:44139 requested a full tablet report, sending...
I20240502 01:54:53.055982 23007 ts_manager.cc:194] Registered new tserver with Master: 29f84c922f9f47f8bd7c360af478d359 (127.9.176.193:42839)
I20240502 01:54:53.057641 23007 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.176.193:48211
W20240502 01:54:53.417918 23197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:54:53.418246 23197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:54:53.418373 23197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:54:53.426357 23197 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:54:53.426512 23197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:54:53.426653 23197 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.194
I20240502 01:54:53.436467 23197 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:43017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.9.176.194
--webserver_port=0
--tserver_master_addrs=127.9.176.254:44139
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.194
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:54:53.437803 23197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:54:53.439594 23197 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:54:53.453377 23202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:53.453855 23203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:53.454600 23205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:54:53.457453 23197 server_base.cc:1034] running on GCE node
I20240502 01:54:53.458168 23197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:54:53.459242 23197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:54:53.460448 23197 hybrid_clock.cc:648] HybridClock initialized: now 1714614893460399 us; error 38 us; skew 500 ppm
I20240502 01:54:53.460779 23197 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:54:53.463404 23197 webserver.cc:460] Webserver started at http://127.9.176.194:33851/ using document root <none> and password file <none>
I20240502 01:54:53.464284 23197 fs_manager.cc:359] Metadata directory not provided
I20240502 01:54:53.464402 23197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:54:53.464715 23197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:54:53.467558 23197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "22b61cbb0f2148db8f62df6a32215c8d"
format_stamp: "Formatted at 2024-05-02 01:54:53 on dist-test-slave-j3hh"
I20240502 01:54:53.468473 23197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "22b61cbb0f2148db8f62df6a32215c8d"
format_stamp: "Formatted at 2024-05-02 01:54:53 on dist-test-slave-j3hh"
I20240502 01:54:53.474437 23197 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.005s	sys 0.000s
I20240502 01:54:53.478499 23211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:54:53.480124 23197 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240502 01:54:53.480324 23197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "22b61cbb0f2148db8f62df6a32215c8d"
format_stamp: "Formatted at 2024-05-02 01:54:53 on dist-test-slave-j3hh"
I20240502 01:54:53.480571 23197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:54:53.499143 23197 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:54:53.500172 23197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:54:53.500489 23197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:54:53.501541 23197 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:54:53.503211 23197 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:54:53.503325 23197 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:54:53.503470 23197 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:54:53.503551 23197 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:54:53.538172 23197 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.194:35853
I20240502 01:54:53.538225 23323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.194:35853 every 8 connection(s)
I20240502 01:54:53.539525 23197 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 01:54:53.542897  9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu as pid 23197
I20240502 01:54:53.543144  9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 01:54:53.547159  9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
/tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.195:0
--local_ip_for_outbound_sockets=127.9.176.195
--webserver_interface=127.9.176.195
--webserver_port=0
--tserver_master_addrs=127.9.176.254:44139
--builtin_ntp_servers=127.9.176.212:43017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:54:53.555230 23324 heartbeater.cc:347] Connected to a master server at 127.9.176.254:44139
I20240502 01:54:53.555646 23324 heartbeater.cc:464] Registering TS with master...
I20240502 01:54:53.556557 23324 heartbeater.cc:511] Master 127.9.176.254:44139 requested a full tablet report, sending...
I20240502 01:54:53.558161 23007 ts_manager.cc:194] Registered new tserver with Master: 22b61cbb0f2148db8f62df6a32215c8d (127.9.176.194:35853)
I20240502 01:54:53.558960 23007 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.176.194:40585
W20240502 01:54:53.911023 23327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:54:53.911379 23327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:54:53.911499 23327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:54:53.919860 23327 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:54:53.920002 23327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:54:53.920178 23327 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.195
I20240502 01:54:53.929826 23327 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:43017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.9.176.195
--webserver_port=0
--tserver_master_addrs=127.9.176.254:44139
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.195
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:54:53.931283 23327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:54:53.933158 23327 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:54:53.945499 23332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:53.946002 23335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:53.946571 23333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:54:53.947321 23327 server_base.cc:1034] running on GCE node
I20240502 01:54:53.948014 23327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:54:53.949074 23327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:54:53.950289 23327 hybrid_clock.cc:648] HybridClock initialized: now 1714614893950242 us; error 48 us; skew 500 ppm
I20240502 01:54:53.950637 23327 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:54:53.953171 23327 webserver.cc:460] Webserver started at http://127.9.176.195:40419/ using document root <none> and password file <none>
I20240502 01:54:53.953979 23327 fs_manager.cc:359] Metadata directory not provided
I20240502 01:54:53.954105 23327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:54:53.954433 23327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:54:53.957321 23327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "218f63e888ff4c6394af93014b59d65f"
format_stamp: "Formatted at 2024-05-02 01:54:53 on dist-test-slave-j3hh"
I20240502 01:54:53.958202 23327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "218f63e888ff4c6394af93014b59d65f"
format_stamp: "Formatted at 2024-05-02 01:54:53 on dist-test-slave-j3hh"
I20240502 01:54:53.964236 23327 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.006s	sys 0.001s
I20240502 01:54:53.968063 23341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:54:53.969473 23327 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20240502 01:54:53.969650 23327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "218f63e888ff4c6394af93014b59d65f"
format_stamp: "Formatted at 2024-05-02 01:54:53 on dist-test-slave-j3hh"
I20240502 01:54:53.969873 23327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:54:53.989825 23327 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:54:53.990758 23327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:54:53.991050 23327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:54:53.992166 23327 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:54:53.993824 23327 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:54:53.993928 23327 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:54:53.994041 23327 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:54:53.994112 23327 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:54:54.030788 23327 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.195:34605
I20240502 01:54:54.030822 23453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.195:34605 every 8 connection(s)
I20240502 01:54:54.032415 23327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 01:54:54.033742  9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu as pid 23327
I20240502 01:54:54.033973  9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 01:54:54.038302  9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
/tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.196:0
--local_ip_for_outbound_sockets=127.9.176.196
--webserver_interface=127.9.176.196
--webserver_port=0
--tserver_master_addrs=127.9.176.254:44139
--builtin_ntp_servers=127.9.176.212:43017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:54:54.049099 23454 heartbeater.cc:347] Connected to a master server at 127.9.176.254:44139
I20240502 01:54:54.049436 23454 heartbeater.cc:464] Registering TS with master...
I20240502 01:54:54.050343 23454 heartbeater.cc:511] Master 127.9.176.254:44139 requested a full tablet report, sending...
I20240502 01:54:54.052376 23007 ts_manager.cc:194] Registered new tserver with Master: 218f63e888ff4c6394af93014b59d65f (127.9.176.195:34605)
I20240502 01:54:54.053787 23007 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.176.195:60727
I20240502 01:54:54.061240 23193 heartbeater.cc:503] Master 127.9.176.254:44139 was elected leader, sending a full tablet report...
W20240502 01:54:54.404664 23457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:54:54.404969 23457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:54:54.405086 23457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:54:54.412781 23457 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:54:54.412925 23457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:54:54.413079 23457 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.196
I20240502 01:54:54.422775 23457 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:43017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.9.176.196
--webserver_port=0
--tserver_master_addrs=127.9.176.254:44139
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.196
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:54:54.424153 23457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:54:54.425895 23457 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:54:54.438699 23463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:54.439124 23465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:54.439792 23462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:54:54.440485 23457 server_base.cc:1034] running on GCE node
I20240502 01:54:54.441095 23457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:54:54.442082 23457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:54:54.443285 23457 hybrid_clock.cc:648] HybridClock initialized: now 1714614894443236 us; error 41 us; skew 500 ppm
I20240502 01:54:54.443660 23457 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:54:54.446082 23457 webserver.cc:460] Webserver started at http://127.9.176.196:44913/ using document root <none> and password file <none>
I20240502 01:54:54.446894 23457 fs_manager.cc:359] Metadata directory not provided
I20240502 01:54:54.447017 23457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:54:54.447368 23457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:54:54.450409 23457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "631566d8d4dc4a43a1a391ea74418418"
format_stamp: "Formatted at 2024-05-02 01:54:54 on dist-test-slave-j3hh"
I20240502 01:54:54.451314 23457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "631566d8d4dc4a43a1a391ea74418418"
format_stamp: "Formatted at 2024-05-02 01:54:54 on dist-test-slave-j3hh"
I20240502 01:54:54.457221 23457 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.000s	sys 0.004s
I20240502 01:54:54.461215 23471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:54:54.462911 23457 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20240502 01:54:54.463100 23457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "631566d8d4dc4a43a1a391ea74418418"
format_stamp: "Formatted at 2024-05-02 01:54:54 on dist-test-slave-j3hh"
I20240502 01:54:54.463332 23457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:54:54.498661 23457 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:54:54.499615 23457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:54:54.499946 23457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:54:54.500933 23457 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:54:54.502485 23457 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:54:54.502574 23457 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:54:54.502698 23457 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:54:54.502780 23457 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:54:54.537415 23457 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.196:45397
I20240502 01:54:54.537544 23583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.196:45397 every 8 connection(s)
I20240502 01:54:54.538904 23457 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 01:54:54.546942  9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu as pid 23457
I20240502 01:54:54.547183  9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 01:54:54.552685 23584 heartbeater.cc:347] Connected to a master server at 127.9.176.254:44139
I20240502 01:54:54.552989 23584 heartbeater.cc:464] Registering TS with master...
I20240502 01:54:54.553818 23584 heartbeater.cc:511] Master 127.9.176.254:44139 requested a full tablet report, sending...
I20240502 01:54:54.555335 23007 ts_manager.cc:194] Registered new tserver with Master: 631566d8d4dc4a43a1a391ea74418418 (127.9.176.196:45397)
I20240502 01:54:54.556285 23007 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.176.196:36889
I20240502 01:54:54.562099 23324 heartbeater.cc:503] Master 127.9.176.254:44139 was elected leader, sending a full tablet report...
I20240502 01:54:54.564071  9923 external_mini_cluster.cc:930] 4 TS(s) registered with all masters
I20240502 01:54:54.580410 23006 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53134:
name: "TestTable"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
owner: "alice"
I20240502 01:54:54.608110 23389 tablet_service.cc:1467] Processing CreateTablet for tablet 6341db525c364256bc0450f381207ad3 (DEFAULT_TABLE table=TestTable [id=b3510111a5b645ea9d3c57df8d2a7a8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:54.609405 23389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6341db525c364256bc0450f381207ad3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:54.620487 23598 tablet_bootstrap.cc:492] T 6341db525c364256bc0450f381207ad3 P 218f63e888ff4c6394af93014b59d65f: Bootstrap starting.
I20240502 01:54:54.623809 23598 tablet_bootstrap.cc:654] T 6341db525c364256bc0450f381207ad3 P 218f63e888ff4c6394af93014b59d65f: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:54.624785 23598 log.cc:824] T 6341db525c364256bc0450f381207ad3 P 218f63e888ff4c6394af93014b59d65f: Log is configured to *not* fsync() on all Append() calls
I20240502 01:54:54.627305 23598 tablet_bootstrap.cc:492] T 6341db525c364256bc0450f381207ad3 P 218f63e888ff4c6394af93014b59d65f: No bootstrap required, opened a new log
I20240502 01:54:54.627626 23598 ts_tablet_manager.cc:1367] T 6341db525c364256bc0450f381207ad3 P 218f63e888ff4c6394af93014b59d65f: Time spent bootstrapping tablet: real 0.007s	user 0.005s	sys 0.000s
I20240502 01:54:54.634020 23598 raft_consensus.cc:348] T 6341db525c364256bc0450f381207ad3 P 218f63e888ff4c6394af93014b59d65f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "218f63e888ff4c6394af93014b59d65f" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 34605 } }
I20240502 01:54:54.634349 23598 raft_consensus.cc:374] T 6341db525c364256bc0450f381207ad3 P 218f63e888ff4c6394af93014b59d65f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:54.634472 23598 raft_consensus.cc:729] T 6341db525c364256bc0450f381207ad3 P 218f63e888ff4c6394af93014b59d65f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 218f63e888ff4c6394af93014b59d65f, State: Initialized, Role: FOLLOWER
I20240502 01:54:54.635074 23598 consensus_queue.cc:260] T 6341db525c364256bc0450f381207ad3 P 218f63e888ff4c6394af93014b59d65f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "218f63e888ff4c6394af93014b59d65f" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 34605 } }
I20240502 01:54:54.635273 23598 raft_consensus.cc:388] T 6341db525c364256bc0450f381207ad3 P 218f63e888ff4c6394af93014b59d65f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:54:54.635401 23598 raft_consensus.cc:482] T 6341db525c364256bc0450f381207ad3 P 218f63e888ff4c6394af93014b59d65f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:54:54.635550 23598 raft_consensus.cc:3037] T 6341db525c364256bc0450f381207ad3 P 218f63e888ff4c6394af93014b59d65f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:54.637552 23598 raft_consensus.cc:504] T 6341db525c364256bc0450f381207ad3 P 218f63e888ff4c6394af93014b59d65f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "218f63e888ff4c6394af93014b59d65f" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 34605 } }
I20240502 01:54:54.638052 23598 leader_election.cc:302] T 6341db525c364256bc0450f381207ad3 P 218f63e888ff4c6394af93014b59d65f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 218f63e888ff4c6394af93014b59d65f; no voters: 
I20240502 01:54:54.638804 23598 leader_election.cc:288] T 6341db525c364256bc0450f381207ad3 P 218f63e888ff4c6394af93014b59d65f [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240502 01:54:54.638947 23600 raft_consensus.cc:2781] T 6341db525c364256bc0450f381207ad3 P 218f63e888ff4c6394af93014b59d65f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:54.639539 23600 raft_consensus.cc:686] T 6341db525c364256bc0450f381207ad3 P 218f63e888ff4c6394af93014b59d65f [term 1 LEADER]: Becoming Leader. State: Replica: 218f63e888ff4c6394af93014b59d65f, State: Running, Role: LEADER
I20240502 01:54:54.640008 23454 heartbeater.cc:503] Master 127.9.176.254:44139 was elected leader, sending a full tablet report...
I20240502 01:54:54.640712 23598 ts_tablet_manager.cc:1398] T 6341db525c364256bc0450f381207ad3 P 218f63e888ff4c6394af93014b59d65f: Time spent starting tablet: real 0.013s	user 0.000s	sys 0.012s
I20240502 01:54:54.640722 23600 consensus_queue.cc:237] T 6341db525c364256bc0450f381207ad3 P 218f63e888ff4c6394af93014b59d65f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "218f63e888ff4c6394af93014b59d65f" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 34605 } }
I20240502 01:54:54.645993 23006 catalog_manager.cc:5484] T 6341db525c364256bc0450f381207ad3 P 218f63e888ff4c6394af93014b59d65f reported cstate change: term changed from 0 to 1, leader changed from <none> to 218f63e888ff4c6394af93014b59d65f (127.9.176.195). New cstate: current_term: 1 leader_uuid: "218f63e888ff4c6394af93014b59d65f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "218f63e888ff4c6394af93014b59d65f" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 34605 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:54.654343  9923 external_mini_cluster.cc:930] 4 TS(s) registered with all masters
I20240502 01:54:54.656181  9923 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 29f84c922f9f47f8bd7c360af478d359 to finish bootstrapping
I20240502 01:54:54.665369  9923 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 22b61cbb0f2148db8f62df6a32215c8d to finish bootstrapping
I20240502 01:54:54.674789  9923 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 218f63e888ff4c6394af93014b59d65f to finish bootstrapping
I20240502 01:54:54.682272  9923 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 631566d8d4dc4a43a1a391ea74418418 to finish bootstrapping
I20240502 01:54:54.690989  9923 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu pid 23327
I20240502 01:54:54.692024 23449 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:54:54.781261  9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu with pid 23327
I20240502 01:54:54.799806  9923 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu pid 23457
I20240502 01:54:54.800980 23579 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:54:54.883550  9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu with pid 23457
I20240502 01:54:58.652801 23007 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53134:
name: "rebalance_test_table_0"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 01:54:58.674925 23259 tablet_service.cc:1467] Processing CreateTablet for tablet 310930a2b5a7416d82706f03e06a5daa (DEFAULT_TABLE table=rebalance_test_table_0 [id=a8cec354879c415095f48afc7deafeba]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:58.675222 23128 tablet_service.cc:1467] Processing CreateTablet for tablet d6ef9d418c534f8ab088edcd3b372856 (DEFAULT_TABLE table=rebalance_test_table_0 [id=a8cec354879c415095f48afc7deafeba]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:58.675174 23127 tablet_service.cc:1467] Processing CreateTablet for tablet 209d6a3dc32f43b1b4d536e02bd0ec73 (DEFAULT_TABLE table=rebalance_test_table_0 [id=a8cec354879c415095f48afc7deafeba]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:58.676383 23259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 310930a2b5a7416d82706f03e06a5daa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:58.676940 23127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 209d6a3dc32f43b1b4d536e02bd0ec73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:58.677421 23128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6ef9d418c534f8ab088edcd3b372856. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:58.688249 23619 tablet_bootstrap.cc:492] T 310930a2b5a7416d82706f03e06a5daa P 22b61cbb0f2148db8f62df6a32215c8d: Bootstrap starting.
I20240502 01:54:58.691002 23620 tablet_bootstrap.cc:492] T 209d6a3dc32f43b1b4d536e02bd0ec73 P 29f84c922f9f47f8bd7c360af478d359: Bootstrap starting.
I20240502 01:54:58.692914 23619 tablet_bootstrap.cc:654] T 310930a2b5a7416d82706f03e06a5daa P 22b61cbb0f2148db8f62df6a32215c8d: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:58.694427 23619 log.cc:824] T 310930a2b5a7416d82706f03e06a5daa P 22b61cbb0f2148db8f62df6a32215c8d: Log is configured to *not* fsync() on all Append() calls
I20240502 01:54:58.694893 23620 tablet_bootstrap.cc:654] T 209d6a3dc32f43b1b4d536e02bd0ec73 P 29f84c922f9f47f8bd7c360af478d359: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:58.696230 23620 log.cc:824] T 209d6a3dc32f43b1b4d536e02bd0ec73 P 29f84c922f9f47f8bd7c360af478d359: Log is configured to *not* fsync() on all Append() calls
I20240502 01:54:58.697657 23619 tablet_bootstrap.cc:492] T 310930a2b5a7416d82706f03e06a5daa P 22b61cbb0f2148db8f62df6a32215c8d: No bootstrap required, opened a new log
I20240502 01:54:58.698343 23619 ts_tablet_manager.cc:1367] T 310930a2b5a7416d82706f03e06a5daa P 22b61cbb0f2148db8f62df6a32215c8d: Time spent bootstrapping tablet: real 0.010s	user 0.005s	sys 0.004s
I20240502 01:54:58.699237 23620 tablet_bootstrap.cc:492] T 209d6a3dc32f43b1b4d536e02bd0ec73 P 29f84c922f9f47f8bd7c360af478d359: No bootstrap required, opened a new log
I20240502 01:54:58.699537 23620 ts_tablet_manager.cc:1367] T 209d6a3dc32f43b1b4d536e02bd0ec73 P 29f84c922f9f47f8bd7c360af478d359: Time spent bootstrapping tablet: real 0.009s	user 0.003s	sys 0.003s
I20240502 01:54:58.708047 23619 raft_consensus.cc:348] T 310930a2b5a7416d82706f03e06a5daa P 22b61cbb0f2148db8f62df6a32215c8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b61cbb0f2148db8f62df6a32215c8d" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 35853 } }
I20240502 01:54:58.708562 23619 raft_consensus.cc:374] T 310930a2b5a7416d82706f03e06a5daa P 22b61cbb0f2148db8f62df6a32215c8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:58.708693 23619 raft_consensus.cc:729] T 310930a2b5a7416d82706f03e06a5daa P 22b61cbb0f2148db8f62df6a32215c8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22b61cbb0f2148db8f62df6a32215c8d, State: Initialized, Role: FOLLOWER
I20240502 01:54:58.708660 23620 raft_consensus.cc:348] T 209d6a3dc32f43b1b4d536e02bd0ec73 P 29f84c922f9f47f8bd7c360af478d359 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29f84c922f9f47f8bd7c360af478d359" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 42839 } }
I20240502 01:54:58.709077 23620 raft_consensus.cc:374] T 209d6a3dc32f43b1b4d536e02bd0ec73 P 29f84c922f9f47f8bd7c360af478d359 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:58.709235 23620 raft_consensus.cc:729] T 209d6a3dc32f43b1b4d536e02bd0ec73 P 29f84c922f9f47f8bd7c360af478d359 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29f84c922f9f47f8bd7c360af478d359, State: Initialized, Role: FOLLOWER
I20240502 01:54:58.709501 23619 consensus_queue.cc:260] T 310930a2b5a7416d82706f03e06a5daa P 22b61cbb0f2148db8f62df6a32215c8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b61cbb0f2148db8f62df6a32215c8d" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 35853 } }
I20240502 01:54:58.709777 23619 raft_consensus.cc:388] T 310930a2b5a7416d82706f03e06a5daa P 22b61cbb0f2148db8f62df6a32215c8d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:54:58.709926 23619 raft_consensus.cc:482] T 310930a2b5a7416d82706f03e06a5daa P 22b61cbb0f2148db8f62df6a32215c8d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:54:58.709924 23620 consensus_queue.cc:260] T 209d6a3dc32f43b1b4d536e02bd0ec73 P 29f84c922f9f47f8bd7c360af478d359 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29f84c922f9f47f8bd7c360af478d359" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 42839 } }
I20240502 01:54:58.710114 23619 raft_consensus.cc:3037] T 310930a2b5a7416d82706f03e06a5daa P 22b61cbb0f2148db8f62df6a32215c8d [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:58.710175 23620 raft_consensus.cc:388] T 209d6a3dc32f43b1b4d536e02bd0ec73 P 29f84c922f9f47f8bd7c360af478d359 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:54:58.710317 23620 raft_consensus.cc:482] T 209d6a3dc32f43b1b4d536e02bd0ec73 P 29f84c922f9f47f8bd7c360af478d359 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:54:58.710505 23620 raft_consensus.cc:3037] T 209d6a3dc32f43b1b4d536e02bd0ec73 P 29f84c922f9f47f8bd7c360af478d359 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:58.713335 23619 raft_consensus.cc:504] T 310930a2b5a7416d82706f03e06a5daa P 22b61cbb0f2148db8f62df6a32215c8d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b61cbb0f2148db8f62df6a32215c8d" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 35853 } }
I20240502 01:54:58.713529 23620 raft_consensus.cc:504] T 209d6a3dc32f43b1b4d536e02bd0ec73 P 29f84c922f9f47f8bd7c360af478d359 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29f84c922f9f47f8bd7c360af478d359" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 42839 } }
I20240502 01:54:58.714025 23619 leader_election.cc:302] T 310930a2b5a7416d82706f03e06a5daa P 22b61cbb0f2148db8f62df6a32215c8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22b61cbb0f2148db8f62df6a32215c8d; no voters: 
I20240502 01:54:58.714012 23620 leader_election.cc:302] T 209d6a3dc32f43b1b4d536e02bd0ec73 P 29f84c922f9f47f8bd7c360af478d359 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 29f84c922f9f47f8bd7c360af478d359; no voters: 
I20240502 01:54:58.714800 23619 leader_election.cc:288] T 310930a2b5a7416d82706f03e06a5daa P 22b61cbb0f2148db8f62df6a32215c8d [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240502 01:54:58.714891 23624 raft_consensus.cc:2781] T 310930a2b5a7416d82706f03e06a5daa P 22b61cbb0f2148db8f62df6a32215c8d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:58.715636 23620 leader_election.cc:288] T 209d6a3dc32f43b1b4d536e02bd0ec73 P 29f84c922f9f47f8bd7c360af478d359 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240502 01:54:58.715778 23623 raft_consensus.cc:2781] T 209d6a3dc32f43b1b4d536e02bd0ec73 P 29f84c922f9f47f8bd7c360af478d359 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:58.716290 23624 raft_consensus.cc:686] T 310930a2b5a7416d82706f03e06a5daa P 22b61cbb0f2148db8f62df6a32215c8d [term 1 LEADER]: Becoming Leader. State: Replica: 22b61cbb0f2148db8f62df6a32215c8d, State: Running, Role: LEADER
I20240502 01:54:58.716886 23624 consensus_queue.cc:237] T 310930a2b5a7416d82706f03e06a5daa P 22b61cbb0f2148db8f62df6a32215c8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b61cbb0f2148db8f62df6a32215c8d" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 35853 } }
I20240502 01:54:58.717814 23619 ts_tablet_manager.cc:1398] T 310930a2b5a7416d82706f03e06a5daa P 22b61cbb0f2148db8f62df6a32215c8d: Time spent starting tablet: real 0.019s	user 0.013s	sys 0.004s
I20240502 01:54:58.720167 23620 ts_tablet_manager.cc:1398] T 209d6a3dc32f43b1b4d536e02bd0ec73 P 29f84c922f9f47f8bd7c360af478d359: Time spent starting tablet: real 0.020s	user 0.018s	sys 0.002s
I20240502 01:54:58.720683 23620 tablet_bootstrap.cc:492] T d6ef9d418c534f8ab088edcd3b372856 P 29f84c922f9f47f8bd7c360af478d359: Bootstrap starting.
I20240502 01:54:58.721460 23623 raft_consensus.cc:686] T 209d6a3dc32f43b1b4d536e02bd0ec73 P 29f84c922f9f47f8bd7c360af478d359 [term 1 LEADER]: Becoming Leader. State: Replica: 29f84c922f9f47f8bd7c360af478d359, State: Running, Role: LEADER
I20240502 01:54:58.721947 23623 consensus_queue.cc:237] T 209d6a3dc32f43b1b4d536e02bd0ec73 P 29f84c922f9f47f8bd7c360af478d359 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29f84c922f9f47f8bd7c360af478d359" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 42839 } }
I20240502 01:54:58.722733 23007 catalog_manager.cc:5484] T 310930a2b5a7416d82706f03e06a5daa P 22b61cbb0f2148db8f62df6a32215c8d reported cstate change: term changed from 0 to 1, leader changed from <none> to 22b61cbb0f2148db8f62df6a32215c8d (127.9.176.194). New cstate: current_term: 1 leader_uuid: "22b61cbb0f2148db8f62df6a32215c8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b61cbb0f2148db8f62df6a32215c8d" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 35853 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:58.723276 23620 tablet_bootstrap.cc:654] T d6ef9d418c534f8ab088edcd3b372856 P 29f84c922f9f47f8bd7c360af478d359: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:58.728258 23006 catalog_manager.cc:5484] T 209d6a3dc32f43b1b4d536e02bd0ec73 P 29f84c922f9f47f8bd7c360af478d359 reported cstate change: term changed from 0 to 1, leader changed from <none> to 29f84c922f9f47f8bd7c360af478d359 (127.9.176.193). New cstate: current_term: 1 leader_uuid: "29f84c922f9f47f8bd7c360af478d359" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29f84c922f9f47f8bd7c360af478d359" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 42839 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:58.733955 23620 tablet_bootstrap.cc:492] T d6ef9d418c534f8ab088edcd3b372856 P 29f84c922f9f47f8bd7c360af478d359: No bootstrap required, opened a new log
I20240502 01:54:58.734148 23620 ts_tablet_manager.cc:1367] T d6ef9d418c534f8ab088edcd3b372856 P 29f84c922f9f47f8bd7c360af478d359: Time spent bootstrapping tablet: real 0.014s	user 0.003s	sys 0.001s
I20240502 01:54:58.735278 23620 raft_consensus.cc:348] T d6ef9d418c534f8ab088edcd3b372856 P 29f84c922f9f47f8bd7c360af478d359 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29f84c922f9f47f8bd7c360af478d359" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 42839 } }
I20240502 01:54:58.735517 23620 raft_consensus.cc:374] T d6ef9d418c534f8ab088edcd3b372856 P 29f84c922f9f47f8bd7c360af478d359 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:58.735659 23620 raft_consensus.cc:729] T d6ef9d418c534f8ab088edcd3b372856 P 29f84c922f9f47f8bd7c360af478d359 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29f84c922f9f47f8bd7c360af478d359, State: Initialized, Role: FOLLOWER
I20240502 01:54:58.735913 23620 consensus_queue.cc:260] T d6ef9d418c534f8ab088edcd3b372856 P 29f84c922f9f47f8bd7c360af478d359 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29f84c922f9f47f8bd7c360af478d359" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 42839 } }
I20240502 01:54:58.736075 23620 raft_consensus.cc:388] T d6ef9d418c534f8ab088edcd3b372856 P 29f84c922f9f47f8bd7c360af478d359 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:54:58.736155 23620 raft_consensus.cc:482] T d6ef9d418c534f8ab088edcd3b372856 P 29f84c922f9f47f8bd7c360af478d359 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:54:58.736248 23620 raft_consensus.cc:3037] T d6ef9d418c534f8ab088edcd3b372856 P 29f84c922f9f47f8bd7c360af478d359 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:58.738368 23620 raft_consensus.cc:504] T d6ef9d418c534f8ab088edcd3b372856 P 29f84c922f9f47f8bd7c360af478d359 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29f84c922f9f47f8bd7c360af478d359" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 42839 } }
I20240502 01:54:58.738643 23620 leader_election.cc:302] T d6ef9d418c534f8ab088edcd3b372856 P 29f84c922f9f47f8bd7c360af478d359 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 29f84c922f9f47f8bd7c360af478d359; no voters: 
I20240502 01:54:58.738823 23620 leader_election.cc:288] T d6ef9d418c534f8ab088edcd3b372856 P 29f84c922f9f47f8bd7c360af478d359 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240502 01:54:58.738940 23623 raft_consensus.cc:2781] T d6ef9d418c534f8ab088edcd3b372856 P 29f84c922f9f47f8bd7c360af478d359 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:58.739295 23620 ts_tablet_manager.cc:1398] T d6ef9d418c534f8ab088edcd3b372856 P 29f84c922f9f47f8bd7c360af478d359: Time spent starting tablet: real 0.005s	user 0.002s	sys 0.000s
I20240502 01:54:58.739509 23623 raft_consensus.cc:686] T d6ef9d418c534f8ab088edcd3b372856 P 29f84c922f9f47f8bd7c360af478d359 [term 1 LEADER]: Becoming Leader. State: Replica: 29f84c922f9f47f8bd7c360af478d359, State: Running, Role: LEADER
I20240502 01:54:58.739790 23623 consensus_queue.cc:237] T d6ef9d418c534f8ab088edcd3b372856 P 29f84c922f9f47f8bd7c360af478d359 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29f84c922f9f47f8bd7c360af478d359" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 42839 } }
I20240502 01:54:58.742148 23006 catalog_manager.cc:5484] T d6ef9d418c534f8ab088edcd3b372856 P 29f84c922f9f47f8bd7c360af478d359 reported cstate change: term changed from 0 to 1, leader changed from <none> to 29f84c922f9f47f8bd7c360af478d359 (127.9.176.193). New cstate: current_term: 1 leader_uuid: "29f84c922f9f47f8bd7c360af478d359" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29f84c922f9f47f8bd7c360af478d359" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 42839 } health_report { overall_health: HEALTHY } } }
I20240502 01:55:00.188863 23006 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53134:
name: "rebalance_test_table_1"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 01:55:00.200279 23259 tablet_service.cc:1467] Processing CreateTablet for tablet 23dff2c6cd704ce6a8853d393dd97119 (DEFAULT_TABLE table=rebalance_test_table_1 [id=80ac4d7f091449898caeebdad7d14f29]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:00.200759 23259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23dff2c6cd704ce6a8853d393dd97119. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:00.203624 23258 tablet_service.cc:1467] Processing CreateTablet for tablet 6213c1863b2048c1a281634dd2693c3e (DEFAULT_TABLE table=rebalance_test_table_1 [id=80ac4d7f091449898caeebdad7d14f29]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:00.203718 23128 tablet_service.cc:1467] Processing CreateTablet for tablet 1f4cf78efc5f479c875536136d17fdb0 (DEFAULT_TABLE table=rebalance_test_table_1 [id=80ac4d7f091449898caeebdad7d14f29]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:00.204075 23258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6213c1863b2048c1a281634dd2693c3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:00.204103 23128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f4cf78efc5f479c875536136d17fdb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:00.205773 23661 tablet_bootstrap.cc:492] T 23dff2c6cd704ce6a8853d393dd97119 P 22b61cbb0f2148db8f62df6a32215c8d: Bootstrap starting.
I20240502 01:55:00.207286 23661 tablet_bootstrap.cc:654] T 23dff2c6cd704ce6a8853d393dd97119 P 22b61cbb0f2148db8f62df6a32215c8d: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:00.209470 23661 tablet_bootstrap.cc:492] T 23dff2c6cd704ce6a8853d393dd97119 P 22b61cbb0f2148db8f62df6a32215c8d: No bootstrap required, opened a new log
I20240502 01:55:00.209709 23661 ts_tablet_manager.cc:1367] T 23dff2c6cd704ce6a8853d393dd97119 P 22b61cbb0f2148db8f62df6a32215c8d: Time spent bootstrapping tablet: real 0.004s	user 0.004s	sys 0.000s
I20240502 01:55:00.210650 23663 tablet_bootstrap.cc:492] T 1f4cf78efc5f479c875536136d17fdb0 P 29f84c922f9f47f8bd7c360af478d359: Bootstrap starting.
I20240502 01:55:00.210844 23661 raft_consensus.cc:348] T 23dff2c6cd704ce6a8853d393dd97119 P 22b61cbb0f2148db8f62df6a32215c8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b61cbb0f2148db8f62df6a32215c8d" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 35853 } }
I20240502 01:55:00.211030 23661 raft_consensus.cc:374] T 23dff2c6cd704ce6a8853d393dd97119 P 22b61cbb0f2148db8f62df6a32215c8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:00.211127 23661 raft_consensus.cc:729] T 23dff2c6cd704ce6a8853d393dd97119 P 22b61cbb0f2148db8f62df6a32215c8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22b61cbb0f2148db8f62df6a32215c8d, State: Initialized, Role: FOLLOWER
I20240502 01:55:00.211392 23661 consensus_queue.cc:260] T 23dff2c6cd704ce6a8853d393dd97119 P 22b61cbb0f2148db8f62df6a32215c8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b61cbb0f2148db8f62df6a32215c8d" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 35853 } }
I20240502 01:55:00.211627 23661 raft_consensus.cc:388] T 23dff2c6cd704ce6a8853d393dd97119 P 22b61cbb0f2148db8f62df6a32215c8d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:55:00.211747 23661 raft_consensus.cc:482] T 23dff2c6cd704ce6a8853d393dd97119 P 22b61cbb0f2148db8f62df6a32215c8d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:55:00.211885 23661 raft_consensus.cc:3037] T 23dff2c6cd704ce6a8853d393dd97119 P 22b61cbb0f2148db8f62df6a32215c8d [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:00.212473 23663 tablet_bootstrap.cc:654] T 1f4cf78efc5f479c875536136d17fdb0 P 29f84c922f9f47f8bd7c360af478d359: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:00.214488 23661 raft_consensus.cc:504] T 23dff2c6cd704ce6a8853d393dd97119 P 22b61cbb0f2148db8f62df6a32215c8d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b61cbb0f2148db8f62df6a32215c8d" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 35853 } }
I20240502 01:55:00.214795 23663 tablet_bootstrap.cc:492] T 1f4cf78efc5f479c875536136d17fdb0 P 29f84c922f9f47f8bd7c360af478d359: No bootstrap required, opened a new log
I20240502 01:55:00.214793 23661 leader_election.cc:302] T 23dff2c6cd704ce6a8853d393dd97119 P 22b61cbb0f2148db8f62df6a32215c8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22b61cbb0f2148db8f62df6a32215c8d; no voters: 
I20240502 01:55:00.214953 23663 ts_tablet_manager.cc:1367] T 1f4cf78efc5f479c875536136d17fdb0 P 29f84c922f9f47f8bd7c360af478d359: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.000s
I20240502 01:55:00.215261 23661 leader_election.cc:288] T 23dff2c6cd704ce6a8853d393dd97119 P 22b61cbb0f2148db8f62df6a32215c8d [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240502 01:55:00.215436 23665 raft_consensus.cc:2781] T 23dff2c6cd704ce6a8853d393dd97119 P 22b61cbb0f2148db8f62df6a32215c8d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:55:00.215863 23663 raft_consensus.cc:348] T 1f4cf78efc5f479c875536136d17fdb0 P 29f84c922f9f47f8bd7c360af478d359 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29f84c922f9f47f8bd7c360af478d359" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 42839 } }
I20240502 01:55:00.215986 23661 ts_tablet_manager.cc:1398] T 23dff2c6cd704ce6a8853d393dd97119 P 22b61cbb0f2148db8f62df6a32215c8d: Time spent starting tablet: real 0.006s	user 0.006s	sys 0.000s
I20240502 01:55:00.216145 23663 raft_consensus.cc:374] T 1f4cf78efc5f479c875536136d17fdb0 P 29f84c922f9f47f8bd7c360af478d359 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:00.216281 23661 tablet_bootstrap.cc:492] T 6213c1863b2048c1a281634dd2693c3e P 22b61cbb0f2148db8f62df6a32215c8d: Bootstrap starting.
I20240502 01:55:00.216301 23663 raft_consensus.cc:729] T 1f4cf78efc5f479c875536136d17fdb0 P 29f84c922f9f47f8bd7c360af478d359 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29f84c922f9f47f8bd7c360af478d359, State: Initialized, Role: FOLLOWER
I20240502 01:55:00.216569 23663 consensus_queue.cc:260] T 1f4cf78efc5f479c875536136d17fdb0 P 29f84c922f9f47f8bd7c360af478d359 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29f84c922f9f47f8bd7c360af478d359" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 42839 } }
I20240502 01:55:00.216764 23663 raft_consensus.cc:388] T 1f4cf78efc5f479c875536136d17fdb0 P 29f84c922f9f47f8bd7c360af478d359 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:55:00.216843 23663 raft_consensus.cc:482] T 1f4cf78efc5f479c875536136d17fdb0 P 29f84c922f9f47f8bd7c360af478d359 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:55:00.216954 23663 raft_consensus.cc:3037] T 1f4cf78efc5f479c875536136d17fdb0 P 29f84c922f9f47f8bd7c360af478d359 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:00.217949 23661 tablet_bootstrap.cc:654] T 6213c1863b2048c1a281634dd2693c3e P 22b61cbb0f2148db8f62df6a32215c8d: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:00.219604 23663 raft_consensus.cc:504] T 1f4cf78efc5f479c875536136d17fdb0 P 29f84c922f9f47f8bd7c360af478d359 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29f84c922f9f47f8bd7c360af478d359" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 42839 } }
I20240502 01:55:00.219825 23665 raft_consensus.cc:686] T 23dff2c6cd704ce6a8853d393dd97119 P 22b61cbb0f2148db8f62df6a32215c8d [term 1 LEADER]: Becoming Leader. State: Replica: 22b61cbb0f2148db8f62df6a32215c8d, State: Running, Role: LEADER
I20240502 01:55:00.219929 23663 leader_election.cc:302] T 1f4cf78efc5f479c875536136d17fdb0 P 29f84c922f9f47f8bd7c360af478d359 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 29f84c922f9f47f8bd7c360af478d359; no voters: 
I20240502 01:55:00.220149 23665 consensus_queue.cc:237] T 23dff2c6cd704ce6a8853d393dd97119 P 22b61cbb0f2148db8f62df6a32215c8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b61cbb0f2148db8f62df6a32215c8d" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 35853 } }
I20240502 01:55:00.220398 23663 leader_election.cc:288] T 1f4cf78efc5f479c875536136d17fdb0 P 29f84c922f9f47f8bd7c360af478d359 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240502 01:55:00.220337 23661 tablet_bootstrap.cc:492] T 6213c1863b2048c1a281634dd2693c3e P 22b61cbb0f2148db8f62df6a32215c8d: No bootstrap required, opened a new log
I20240502 01:55:00.220526 23669 raft_consensus.cc:2781] T 1f4cf78efc5f479c875536136d17fdb0 P 29f84c922f9f47f8bd7c360af478d359 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:55:00.220587 23661 ts_tablet_manager.cc:1367] T 6213c1863b2048c1a281634dd2693c3e P 22b61cbb0f2148db8f62df6a32215c8d: Time spent bootstrapping tablet: real 0.004s	user 0.004s	sys 0.000s
I20240502 01:55:00.221628 23661 raft_consensus.cc:348] T 6213c1863b2048c1a281634dd2693c3e P 22b61cbb0f2148db8f62df6a32215c8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b61cbb0f2148db8f62df6a32215c8d" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 35853 } }
I20240502 01:55:00.221972 23661 raft_consensus.cc:374] T 6213c1863b2048c1a281634dd2693c3e P 22b61cbb0f2148db8f62df6a32215c8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:00.222133 23661 raft_consensus.cc:729] T 6213c1863b2048c1a281634dd2693c3e P 22b61cbb0f2148db8f62df6a32215c8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22b61cbb0f2148db8f62df6a32215c8d, State: Initialized, Role: FOLLOWER
I20240502 01:55:00.222545 23661 consensus_queue.cc:260] T 6213c1863b2048c1a281634dd2693c3e P 22b61cbb0f2148db8f62df6a32215c8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b61cbb0f2148db8f62df6a32215c8d" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 35853 } }
I20240502 01:55:00.222842 23661 raft_consensus.cc:388] T 6213c1863b2048c1a281634dd2693c3e P 22b61cbb0f2148db8f62df6a32215c8d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:55:00.223024 23661 raft_consensus.cc:482] T 6213c1863b2048c1a281634dd2693c3e P 22b61cbb0f2148db8f62df6a32215c8d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:55:00.223199 23661 raft_consensus.cc:3037] T 6213c1863b2048c1a281634dd2693c3e P 22b61cbb0f2148db8f62df6a32215c8d [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:00.227257 23661 raft_consensus.cc:504] T 6213c1863b2048c1a281634dd2693c3e P 22b61cbb0f2148db8f62df6a32215c8d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b61cbb0f2148db8f62df6a32215c8d" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 35853 } }
I20240502 01:55:00.227809 23661 leader_election.cc:302] T 6213c1863b2048c1a281634dd2693c3e P 22b61cbb0f2148db8f62df6a32215c8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22b61cbb0f2148db8f62df6a32215c8d; no voters: 
I20240502 01:55:00.228345 23667 raft_consensus.cc:2781] T 6213c1863b2048c1a281634dd2693c3e P 22b61cbb0f2148db8f62df6a32215c8d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:55:00.228531 23667 raft_consensus.cc:686] T 6213c1863b2048c1a281634dd2693c3e P 22b61cbb0f2148db8f62df6a32215c8d [term 1 LEADER]: Becoming Leader. State: Replica: 22b61cbb0f2148db8f62df6a32215c8d, State: Running, Role: LEADER
I20240502 01:55:00.228879 23667 consensus_queue.cc:237] T 6213c1863b2048c1a281634dd2693c3e P 22b61cbb0f2148db8f62df6a32215c8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b61cbb0f2148db8f62df6a32215c8d" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 35853 } }
I20240502 01:55:00.230006 23661 leader_election.cc:288] T 6213c1863b2048c1a281634dd2693c3e P 22b61cbb0f2148db8f62df6a32215c8d [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240502 01:55:00.230616 23661 ts_tablet_manager.cc:1398] T 6213c1863b2048c1a281634dd2693c3e P 22b61cbb0f2148db8f62df6a32215c8d: Time spent starting tablet: real 0.010s	user 0.003s	sys 0.004s
I20240502 01:55:00.231603 23663 ts_tablet_manager.cc:1398] T 1f4cf78efc5f479c875536136d17fdb0 P 29f84c922f9f47f8bd7c360af478d359: Time spent starting tablet: real 0.016s	user 0.004s	sys 0.011s
I20240502 01:55:00.232909 23669 raft_consensus.cc:686] T 1f4cf78efc5f479c875536136d17fdb0 P 29f84c922f9f47f8bd7c360af478d359 [term 1 LEADER]: Becoming Leader. State: Replica: 29f84c922f9f47f8bd7c360af478d359, State: Running, Role: LEADER
I20240502 01:55:00.233245 23669 consensus_queue.cc:237] T 1f4cf78efc5f479c875536136d17fdb0 P 29f84c922f9f47f8bd7c360af478d359 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29f84c922f9f47f8bd7c360af478d359" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 42839 } }
I20240502 01:55:00.234251 23006 catalog_manager.cc:5484] T 23dff2c6cd704ce6a8853d393dd97119 P 22b61cbb0f2148db8f62df6a32215c8d reported cstate change: term changed from 0 to 1, leader changed from <none> to 22b61cbb0f2148db8f62df6a32215c8d (127.9.176.194). New cstate: current_term: 1 leader_uuid: "22b61cbb0f2148db8f62df6a32215c8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b61cbb0f2148db8f62df6a32215c8d" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 35853 } health_report { overall_health: HEALTHY } } }
I20240502 01:55:00.235606 23007 catalog_manager.cc:5484] T 1f4cf78efc5f479c875536136d17fdb0 P 29f84c922f9f47f8bd7c360af478d359 reported cstate change: term changed from 0 to 1, leader changed from <none> to 29f84c922f9f47f8bd7c360af478d359 (127.9.176.193). New cstate: current_term: 1 leader_uuid: "29f84c922f9f47f8bd7c360af478d359" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29f84c922f9f47f8bd7c360af478d359" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 42839 } health_report { overall_health: HEALTHY } } }
I20240502 01:55:00.239956 23007 catalog_manager.cc:5484] T 6213c1863b2048c1a281634dd2693c3e P 22b61cbb0f2148db8f62df6a32215c8d reported cstate change: term changed from 0 to 1, leader changed from <none> to 22b61cbb0f2148db8f62df6a32215c8d (127.9.176.194). New cstate: current_term: 1 leader_uuid: "22b61cbb0f2148db8f62df6a32215c8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b61cbb0f2148db8f62df6a32215c8d" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 35853 } health_report { overall_health: HEALTHY } } }
I20240502 01:55:00.757474 23007 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53134:
name: "rebalance_test_table_2"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 01:55:00.769671 23128 tablet_service.cc:1467] Processing CreateTablet for tablet 49eddfad6f6e4247a83e82964e8a9470 (DEFAULT_TABLE table=rebalance_test_table_2 [id=6256f1b0e93c4aca801bfb7d7031a5af]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:00.770218 23128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49eddfad6f6e4247a83e82964e8a9470. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:00.770295 23258 tablet_service.cc:1467] Processing CreateTablet for tablet 33acbbf5980c41adbc6b2ac5d5ddb0ae (DEFAULT_TABLE table=rebalance_test_table_2 [id=6256f1b0e93c4aca801bfb7d7031a5af]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:00.770493 23127 tablet_service.cc:1467] Processing CreateTablet for tablet b9422d7f0ab644979df332552c1e9d14 (DEFAULT_TABLE table=rebalance_test_table_2 [id=6256f1b0e93c4aca801bfb7d7031a5af]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:00.770902 23258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33acbbf5980c41adbc6b2ac5d5ddb0ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:00.770956 23127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9422d7f0ab644979df332552c1e9d14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:00.776062 23702 tablet_bootstrap.cc:492] T b9422d7f0ab644979df332552c1e9d14 P 29f84c922f9f47f8bd7c360af478d359: Bootstrap starting.
I20240502 01:55:00.777698 23702 tablet_bootstrap.cc:654] T b9422d7f0ab644979df332552c1e9d14 P 29f84c922f9f47f8bd7c360af478d359: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:00.779642 23702 tablet_bootstrap.cc:492] T b9422d7f0ab644979df332552c1e9d14 P 29f84c922f9f47f8bd7c360af478d359: No bootstrap required, opened a new log
I20240502 01:55:00.779814 23702 ts_tablet_manager.cc:1367] T b9422d7f0ab644979df332552c1e9d14 P 29f84c922f9f47f8bd7c360af478d359: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.001s
I20240502 01:55:00.780440 23702 raft_consensus.cc:348] T b9422d7f0ab644979df332552c1e9d14 P 29f84c922f9f47f8bd7c360af478d359 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29f84c922f9f47f8bd7c360af478d359" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 42839 } }
I20240502 01:55:00.780606 23702 raft_consensus.cc:374] T b9422d7f0ab644979df332552c1e9d14 P 29f84c922f9f47f8bd7c360af478d359 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:00.780709 23702 raft_consensus.cc:729] T b9422d7f0ab644979df332552c1e9d14 P 29f84c922f9f47f8bd7c360af478d359 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29f84c922f9f47f8bd7c360af478d359, State: Initialized, Role: FOLLOWER
I20240502 01:55:00.780982 23702 consensus_queue.cc:260] T b9422d7f0ab644979df332552c1e9d14 P 29f84c922f9f47f8bd7c360af478d359 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29f84c922f9f47f8bd7c360af478d359" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 42839 } }
I20240502 01:55:00.781158 23702 raft_consensus.cc:388] T b9422d7f0ab644979df332552c1e9d14 P 29f84c922f9f47f8bd7c360af478d359 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:55:00.781252 23702 raft_consensus.cc:482] T b9422d7f0ab644979df332552c1e9d14 P 29f84c922f9f47f8bd7c360af478d359 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:55:00.781369 23702 raft_consensus.cc:3037] T b9422d7f0ab644979df332552c1e9d14 P 29f84c922f9f47f8bd7c360af478d359 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:00.783625 23702 raft_consensus.cc:504] T b9422d7f0ab644979df332552c1e9d14 P 29f84c922f9f47f8bd7c360af478d359 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29f84c922f9f47f8bd7c360af478d359" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 42839 } }
I20240502 01:55:00.783890 23702 leader_election.cc:302] T b9422d7f0ab644979df332552c1e9d14 P 29f84c922f9f47f8bd7c360af478d359 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 29f84c922f9f47f8bd7c360af478d359; no voters: 
I20240502 01:55:00.784358 23702 leader_election.cc:288] T b9422d7f0ab644979df332552c1e9d14 P 29f84c922f9f47f8bd7c360af478d359 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240502 01:55:00.784528 23705 raft_consensus.cc:2781] T b9422d7f0ab644979df332552c1e9d14 P 29f84c922f9f47f8bd7c360af478d359 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:55:00.785354 23702 ts_tablet_manager.cc:1398] T b9422d7f0ab644979df332552c1e9d14 P 29f84c922f9f47f8bd7c360af478d359: Time spent starting tablet: real 0.005s	user 0.002s	sys 0.004s
I20240502 01:55:00.785569 23703 tablet_bootstrap.cc:492] T 33acbbf5980c41adbc6b2ac5d5ddb0ae P 22b61cbb0f2148db8f62df6a32215c8d: Bootstrap starting.
I20240502 01:55:00.785773 23702 tablet_bootstrap.cc:492] T 49eddfad6f6e4247a83e82964e8a9470 P 29f84c922f9f47f8bd7c360af478d359: Bootstrap starting.
I20240502 01:55:00.787519 23703 tablet_bootstrap.cc:654] T 33acbbf5980c41adbc6b2ac5d5ddb0ae P 22b61cbb0f2148db8f62df6a32215c8d: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:00.788163 23702 tablet_bootstrap.cc:654] T 49eddfad6f6e4247a83e82964e8a9470 P 29f84c922f9f47f8bd7c360af478d359: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:00.789402 23703 tablet_bootstrap.cc:492] T 33acbbf5980c41adbc6b2ac5d5ddb0ae P 22b61cbb0f2148db8f62df6a32215c8d: No bootstrap required, opened a new log
I20240502 01:55:00.789657 23703 ts_tablet_manager.cc:1367] T 33acbbf5980c41adbc6b2ac5d5ddb0ae P 22b61cbb0f2148db8f62df6a32215c8d: Time spent bootstrapping tablet: real 0.004s	user 0.001s	sys 0.003s
I20240502 01:55:00.789873 23705 raft_consensus.cc:686] T b9422d7f0ab644979df332552c1e9d14 P 29f84c922f9f47f8bd7c360af478d359 [term 1 LEADER]: Becoming Leader. State: Replica: 29f84c922f9f47f8bd7c360af478d359, State: Running, Role: LEADER
I20240502 01:55:00.790241 23705 consensus_queue.cc:237] T b9422d7f0ab644979df332552c1e9d14 P 29f84c922f9f47f8bd7c360af478d359 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29f84c922f9f47f8bd7c360af478d359" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 42839 } }
I20240502 01:55:00.790468 23703 raft_consensus.cc:348] T 33acbbf5980c41adbc6b2ac5d5ddb0ae P 22b61cbb0f2148db8f62df6a32215c8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b61cbb0f2148db8f62df6a32215c8d" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 35853 } }
I20240502 01:55:00.790721 23703 raft_consensus.cc:374] T 33acbbf5980c41adbc6b2ac5d5ddb0ae P 22b61cbb0f2148db8f62df6a32215c8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:00.790860 23702 tablet_bootstrap.cc:492] T 49eddfad6f6e4247a83e82964e8a9470 P 29f84c922f9f47f8bd7c360af478d359: No bootstrap required, opened a new log
I20240502 01:55:00.790879 23703 raft_consensus.cc:729] T 33acbbf5980c41adbc6b2ac5d5ddb0ae P 22b61cbb0f2148db8f62df6a32215c8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22b61cbb0f2148db8f62df6a32215c8d, State: Initialized, Role: FOLLOWER
I20240502 01:55:00.791076 23702 ts_tablet_manager.cc:1367] T 49eddfad6f6e4247a83e82964e8a9470 P 29f84c922f9f47f8bd7c360af478d359: Time spent bootstrapping tablet: real 0.005s	user 0.005s	sys 0.000s
I20240502 01:55:00.791136 23703 consensus_queue.cc:260] T 33acbbf5980c41adbc6b2ac5d5ddb0ae P 22b61cbb0f2148db8f62df6a32215c8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b61cbb0f2148db8f62df6a32215c8d" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 35853 } }
I20240502 01:55:00.791411 23703 raft_consensus.cc:388] T 33acbbf5980c41adbc6b2ac5d5ddb0ae P 22b61cbb0f2148db8f62df6a32215c8d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:55:00.791548 23703 raft_consensus.cc:482] T 33acbbf5980c41adbc6b2ac5d5ddb0ae P 22b61cbb0f2148db8f62df6a32215c8d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:55:00.791756 23703 raft_consensus.cc:3037] T 33acbbf5980c41adbc6b2ac5d5ddb0ae P 22b61cbb0f2148db8f62df6a32215c8d [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:00.792296 23702 raft_consensus.cc:348] T 49eddfad6f6e4247a83e82964e8a9470 P 29f84c922f9f47f8bd7c360af478d359 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29f84c922f9f47f8bd7c360af478d359" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 42839 } }
I20240502 01:55:00.792618 23702 raft_consensus.cc:374] T 49eddfad6f6e4247a83e82964e8a9470 P 29f84c922f9f47f8bd7c360af478d359 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:00.792763 23702 raft_consensus.cc:729] T 49eddfad6f6e4247a83e82964e8a9470 P 29f84c922f9f47f8bd7c360af478d359 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29f84c922f9f47f8bd7c360af478d359, State: Initialized, Role: FOLLOWER
I20240502 01:55:00.793182 23702 consensus_queue.cc:260] T 49eddfad6f6e4247a83e82964e8a9470 P 29f84c922f9f47f8bd7c360af478d359 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29f84c922f9f47f8bd7c360af478d359" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 42839 } }
I20240502 01:55:00.793483 23702 raft_consensus.cc:388] T 49eddfad6f6e4247a83e82964e8a9470 P 29f84c922f9f47f8bd7c360af478d359 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:55:00.793668 23702 raft_consensus.cc:482] T 49eddfad6f6e4247a83e82964e8a9470 P 29f84c922f9f47f8bd7c360af478d359 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:55:00.793853 23702 raft_consensus.cc:3037] T 49eddfad6f6e4247a83e82964e8a9470 P 29f84c922f9f47f8bd7c360af478d359 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:00.794014 23703 raft_consensus.cc:504] T 33acbbf5980c41adbc6b2ac5d5ddb0ae P 22b61cbb0f2148db8f62df6a32215c8d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b61cbb0f2148db8f62df6a32215c8d" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 35853 } }
I20240502 01:55:00.794286 23703 leader_election.cc:302] T 33acbbf5980c41adbc6b2ac5d5ddb0ae P 22b61cbb0f2148db8f62df6a32215c8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22b61cbb0f2148db8f62df6a32215c8d; no voters: 
I20240502 01:55:00.794538 23703 leader_election.cc:288] T 33acbbf5980c41adbc6b2ac5d5ddb0ae P 22b61cbb0f2148db8f62df6a32215c8d [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240502 01:55:00.794602 23665 raft_consensus.cc:2781] T 33acbbf5980c41adbc6b2ac5d5ddb0ae P 22b61cbb0f2148db8f62df6a32215c8d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:55:00.798005 23702 raft_consensus.cc:504] T 49eddfad6f6e4247a83e82964e8a9470 P 29f84c922f9f47f8bd7c360af478d359 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29f84c922f9f47f8bd7c360af478d359" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 42839 } }
I20240502 01:55:00.798102 23665 raft_consensus.cc:686] T 33acbbf5980c41adbc6b2ac5d5ddb0ae P 22b61cbb0f2148db8f62df6a32215c8d [term 1 LEADER]: Becoming Leader. State: Replica: 22b61cbb0f2148db8f62df6a32215c8d, State: Running, Role: LEADER
I20240502 01:55:00.798449 23702 leader_election.cc:302] T 49eddfad6f6e4247a83e82964e8a9470 P 29f84c922f9f47f8bd7c360af478d359 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 29f84c922f9f47f8bd7c360af478d359; no voters: 
I20240502 01:55:00.798524 23665 consensus_queue.cc:237] T 33acbbf5980c41adbc6b2ac5d5ddb0ae P 22b61cbb0f2148db8f62df6a32215c8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b61cbb0f2148db8f62df6a32215c8d" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 35853 } }
I20240502 01:55:00.798857 23702 leader_election.cc:288] T 49eddfad6f6e4247a83e82964e8a9470 P 29f84c922f9f47f8bd7c360af478d359 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240502 01:55:00.799150 23707 raft_consensus.cc:2781] T 49eddfad6f6e4247a83e82964e8a9470 P 29f84c922f9f47f8bd7c360af478d359 [term 1 FOLLOWER]: Leader election won for term 1
W20240502 01:55:00.799598 23194 tablet.cc:2369] T b9422d7f0ab644979df332552c1e9d14 P 29f84c922f9f47f8bd7c360af478d359: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:55:00.799634 23707 raft_consensus.cc:686] T 49eddfad6f6e4247a83e82964e8a9470 P 29f84c922f9f47f8bd7c360af478d359 [term 1 LEADER]: Becoming Leader. State: Replica: 29f84c922f9f47f8bd7c360af478d359, State: Running, Role: LEADER
I20240502 01:55:00.799974 23707 consensus_queue.cc:237] T 49eddfad6f6e4247a83e82964e8a9470 P 29f84c922f9f47f8bd7c360af478d359 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29f84c922f9f47f8bd7c360af478d359" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 42839 } }
I20240502 01:55:00.801260 23702 ts_tablet_manager.cc:1398] T 49eddfad6f6e4247a83e82964e8a9470 P 29f84c922f9f47f8bd7c360af478d359: Time spent starting tablet: real 0.010s	user 0.007s	sys 0.000s
I20240502 01:55:00.802423 23703 ts_tablet_manager.cc:1398] T 33acbbf5980c41adbc6b2ac5d5ddb0ae P 22b61cbb0f2148db8f62df6a32215c8d: Time spent starting tablet: real 0.013s	user 0.006s	sys 0.005s
I20240502 01:55:00.806174 23007 catalog_manager.cc:5484] T 33acbbf5980c41adbc6b2ac5d5ddb0ae P 22b61cbb0f2148db8f62df6a32215c8d reported cstate change: term changed from 0 to 1, leader changed from <none> to 22b61cbb0f2148db8f62df6a32215c8d (127.9.176.194). New cstate: current_term: 1 leader_uuid: "22b61cbb0f2148db8f62df6a32215c8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b61cbb0f2148db8f62df6a32215c8d" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 35853 } health_report { overall_health: HEALTHY } } }
I20240502 01:55:00.808740 23006 catalog_manager.cc:5484] T b9422d7f0ab644979df332552c1e9d14 P 29f84c922f9f47f8bd7c360af478d359 reported cstate change: term changed from 0 to 1, leader changed from <none> to 29f84c922f9f47f8bd7c360af478d359 (127.9.176.193). New cstate: current_term: 1 leader_uuid: "29f84c922f9f47f8bd7c360af478d359" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29f84c922f9f47f8bd7c360af478d359" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 42839 } health_report { overall_health: HEALTHY } } }
I20240502 01:55:00.814527 23006 catalog_manager.cc:5484] T 49eddfad6f6e4247a83e82964e8a9470 P 29f84c922f9f47f8bd7c360af478d359 reported cstate change: term changed from 0 to 1, leader changed from <none> to 29f84c922f9f47f8bd7c360af478d359 (127.9.176.193). New cstate: current_term: 1 leader_uuid: "29f84c922f9f47f8bd7c360af478d359" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29f84c922f9f47f8bd7c360af478d359" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 42839 } health_report { overall_health: HEALTHY } } }
I20240502 01:55:02.248469 23006 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53134:
name: "rebalance_test_table_3"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 01:55:02.256999 23258 tablet_service.cc:1467] Processing CreateTablet for tablet 72b01a904eab46e7865356ab5358acb6 (DEFAULT_TABLE table=rebalance_test_table_3 [id=88b8e1b8224046a2bcff6f69c6233906]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:02.257134 23259 tablet_service.cc:1467] Processing CreateTablet for tablet 0e2012468bb94a30a1d62cb58a98fc8c (DEFAULT_TABLE table=rebalance_test_table_3 [id=88b8e1b8224046a2bcff6f69c6233906]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:02.257539 23258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72b01a904eab46e7865356ab5358acb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:02.257771 23259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e2012468bb94a30a1d62cb58a98fc8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:02.258052 23128 tablet_service.cc:1467] Processing CreateTablet for tablet 5c23236383d646d29f39a5c18938fd24 (DEFAULT_TABLE table=rebalance_test_table_3 [id=88b8e1b8224046a2bcff6f69c6233906]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:02.258438 23128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c23236383d646d29f39a5c18938fd24. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:02.264636 23744 tablet_bootstrap.cc:492] T 0e2012468bb94a30a1d62cb58a98fc8c P 22b61cbb0f2148db8f62df6a32215c8d: Bootstrap starting.
I20240502 01:55:02.267158 23744 tablet_bootstrap.cc:654] T 0e2012468bb94a30a1d62cb58a98fc8c P 22b61cbb0f2148db8f62df6a32215c8d: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:02.268020 23743 tablet_bootstrap.cc:492] T 5c23236383d646d29f39a5c18938fd24 P 29f84c922f9f47f8bd7c360af478d359: Bootstrap starting.
I20240502 01:55:02.269172 23744 tablet_bootstrap.cc:492] T 0e2012468bb94a30a1d62cb58a98fc8c P 22b61cbb0f2148db8f62df6a32215c8d: No bootstrap required, opened a new log
I20240502 01:55:02.269294 23744 ts_tablet_manager.cc:1367] T 0e2012468bb94a30a1d62cb58a98fc8c P 22b61cbb0f2148db8f62df6a32215c8d: Time spent bootstrapping tablet: real 0.005s	user 0.000s	sys 0.004s
I20240502 01:55:02.269563 23743 tablet_bootstrap.cc:654] T 5c23236383d646d29f39a5c18938fd24 P 29f84c922f9f47f8bd7c360af478d359: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:02.269984 23744 raft_consensus.cc:348] T 0e2012468bb94a30a1d62cb58a98fc8c P 22b61cbb0f2148db8f62df6a32215c8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b61cbb0f2148db8f62df6a32215c8d" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 35853 } }
I20240502 01:55:02.270134 23744 raft_consensus.cc:374] T 0e2012468bb94a30a1d62cb58a98fc8c P 22b61cbb0f2148db8f62df6a32215c8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:02.270264 23744 raft_consensus.cc:729] T 0e2012468bb94a30a1d62cb58a98fc8c P 22b61cbb0f2148db8f62df6a32215c8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22b61cbb0f2148db8f62df6a32215c8d, State: Initialized, Role: FOLLOWER
I20240502 01:55:02.270541 23744 consensus_queue.cc:260] T 0e2012468bb94a30a1d62cb58a98fc8c P 22b61cbb0f2148db8f62df6a32215c8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b61cbb0f2148db8f62df6a32215c8d" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 35853 } }
I20240502 01:55:02.270763 23744 raft_consensus.cc:388] T 0e2012468bb94a30a1d62cb58a98fc8c P 22b61cbb0f2148db8f62df6a32215c8d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:55:02.271040 23744 raft_consensus.cc:482] T 0e2012468bb94a30a1d62cb58a98fc8c P 22b61cbb0f2148db8f62df6a32215c8d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:55:02.271158 23744 raft_consensus.cc:3037] T 0e2012468bb94a30a1d62cb58a98fc8c P 22b61cbb0f2148db8f62df6a32215c8d [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:02.271360 23743 tablet_bootstrap.cc:492] T 5c23236383d646d29f39a5c18938fd24 P 29f84c922f9f47f8bd7c360af478d359: No bootstrap required, opened a new log
I20240502 01:55:02.271492 23743 ts_tablet_manager.cc:1367] T 5c23236383d646d29f39a5c18938fd24 P 29f84c922f9f47f8bd7c360af478d359: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.000s
I20240502 01:55:02.272219 23743 raft_consensus.cc:348] T 5c23236383d646d29f39a5c18938fd24 P 29f84c922f9f47f8bd7c360af478d359 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29f84c922f9f47f8bd7c360af478d359" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 42839 } }
I20240502 01:55:02.272367 23743 raft_consensus.cc:374] T 5c23236383d646d29f39a5c18938fd24 P 29f84c922f9f47f8bd7c360af478d359 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:02.272436 23743 raft_consensus.cc:729] T 5c23236383d646d29f39a5c18938fd24 P 29f84c922f9f47f8bd7c360af478d359 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29f84c922f9f47f8bd7c360af478d359, State: Initialized, Role: FOLLOWER
I20240502 01:55:02.272653 23743 consensus_queue.cc:260] T 5c23236383d646d29f39a5c18938fd24 P 29f84c922f9f47f8bd7c360af478d359 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29f84c922f9f47f8bd7c360af478d359" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 42839 } }
I20240502 01:55:02.272819 23743 raft_consensus.cc:388] T 5c23236383d646d29f39a5c18938fd24 P 29f84c922f9f47f8bd7c360af478d359 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:55:02.272899 23743 raft_consensus.cc:482] T 5c23236383d646d29f39a5c18938fd24 P 29f84c922f9f47f8bd7c360af478d359 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:55:02.273020 23743 raft_consensus.cc:3037] T 5c23236383d646d29f39a5c18938fd24 P 29f84c922f9f47f8bd7c360af478d359 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:02.273488 23744 raft_consensus.cc:504] T 0e2012468bb94a30a1d62cb58a98fc8c P 22b61cbb0f2148db8f62df6a32215c8d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b61cbb0f2148db8f62df6a32215c8d" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 35853 } }
I20240502 01:55:02.273736 23744 leader_election.cc:302] T 0e2012468bb94a30a1d62cb58a98fc8c P 22b61cbb0f2148db8f62df6a32215c8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22b61cbb0f2148db8f62df6a32215c8d; no voters: 
I20240502 01:55:02.274176 23744 leader_election.cc:288] T 0e2012468bb94a30a1d62cb58a98fc8c P 22b61cbb0f2148db8f62df6a32215c8d [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240502 01:55:02.274307 23747 raft_consensus.cc:2781] T 0e2012468bb94a30a1d62cb58a98fc8c P 22b61cbb0f2148db8f62df6a32215c8d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:55:02.275166 23743 raft_consensus.cc:504] T 5c23236383d646d29f39a5c18938fd24 P 29f84c922f9f47f8bd7c360af478d359 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29f84c922f9f47f8bd7c360af478d359" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 42839 } }
I20240502 01:55:02.275401 23743 leader_election.cc:302] T 5c23236383d646d29f39a5c18938fd24 P 29f84c922f9f47f8bd7c360af478d359 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 29f84c922f9f47f8bd7c360af478d359; no voters: 
I20240502 01:55:02.275414 23744 ts_tablet_manager.cc:1398] T 0e2012468bb94a30a1d62cb58a98fc8c P 22b61cbb0f2148db8f62df6a32215c8d: Time spent starting tablet: real 0.006s	user 0.000s	sys 0.005s
I20240502 01:55:02.275779 23744 tablet_bootstrap.cc:492] T 72b01a904eab46e7865356ab5358acb6 P 22b61cbb0f2148db8f62df6a32215c8d: Bootstrap starting.
I20240502 01:55:02.277685 23743 leader_election.cc:288] T 5c23236383d646d29f39a5c18938fd24 P 29f84c922f9f47f8bd7c360af478d359 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240502 01:55:02.277810 23750 raft_consensus.cc:2781] T 5c23236383d646d29f39a5c18938fd24 P 29f84c922f9f47f8bd7c360af478d359 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:55:02.278234 23744 tablet_bootstrap.cc:654] T 72b01a904eab46e7865356ab5358acb6 P 22b61cbb0f2148db8f62df6a32215c8d: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:02.281487 23747 raft_consensus.cc:686] T 0e2012468bb94a30a1d62cb58a98fc8c P 22b61cbb0f2148db8f62df6a32215c8d [term 1 LEADER]: Becoming Leader. State: Replica: 22b61cbb0f2148db8f62df6a32215c8d, State: Running, Role: LEADER
I20240502 01:55:02.282027 23747 consensus_queue.cc:237] T 0e2012468bb94a30a1d62cb58a98fc8c P 22b61cbb0f2148db8f62df6a32215c8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b61cbb0f2148db8f62df6a32215c8d" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 35853 } }
I20240502 01:55:02.284543 23744 tablet_bootstrap.cc:492] T 72b01a904eab46e7865356ab5358acb6 P 22b61cbb0f2148db8f62df6a32215c8d: No bootstrap required, opened a new log
I20240502 01:55:02.284932 23744 ts_tablet_manager.cc:1367] T 72b01a904eab46e7865356ab5358acb6 P 22b61cbb0f2148db8f62df6a32215c8d: Time spent bootstrapping tablet: real 0.009s	user 0.004s	sys 0.004s
I20240502 01:55:02.285104 23743 ts_tablet_manager.cc:1398] T 5c23236383d646d29f39a5c18938fd24 P 29f84c922f9f47f8bd7c360af478d359: Time spent starting tablet: real 0.013s	user 0.009s	sys 0.004s
I20240502 01:55:02.285566 23750 raft_consensus.cc:686] T 5c23236383d646d29f39a5c18938fd24 P 29f84c922f9f47f8bd7c360af478d359 [term 1 LEADER]: Becoming Leader. State: Replica: 29f84c922f9f47f8bd7c360af478d359, State: Running, Role: LEADER
I20240502 01:55:02.285959 23750 consensus_queue.cc:237] T 5c23236383d646d29f39a5c18938fd24 P 29f84c922f9f47f8bd7c360af478d359 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29f84c922f9f47f8bd7c360af478d359" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 42839 } }
I20240502 01:55:02.286231 23744 raft_consensus.cc:348] T 72b01a904eab46e7865356ab5358acb6 P 22b61cbb0f2148db8f62df6a32215c8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b61cbb0f2148db8f62df6a32215c8d" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 35853 } }
I20240502 01:55:02.286553 23744 raft_consensus.cc:374] T 72b01a904eab46e7865356ab5358acb6 P 22b61cbb0f2148db8f62df6a32215c8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:02.286736 23744 raft_consensus.cc:729] T 72b01a904eab46e7865356ab5358acb6 P 22b61cbb0f2148db8f62df6a32215c8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22b61cbb0f2148db8f62df6a32215c8d, State: Initialized, Role: FOLLOWER
I20240502 01:55:02.287138 23744 consensus_queue.cc:260] T 72b01a904eab46e7865356ab5358acb6 P 22b61cbb0f2148db8f62df6a32215c8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b61cbb0f2148db8f62df6a32215c8d" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 35853 } }
I20240502 01:55:02.287362 23744 raft_consensus.cc:388] T 72b01a904eab46e7865356ab5358acb6 P 22b61cbb0f2148db8f62df6a32215c8d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:55:02.287484 23744 raft_consensus.cc:482] T 72b01a904eab46e7865356ab5358acb6 P 22b61cbb0f2148db8f62df6a32215c8d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:55:02.287691 23744 raft_consensus.cc:3037] T 72b01a904eab46e7865356ab5358acb6 P 22b61cbb0f2148db8f62df6a32215c8d [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:02.289039 23006 catalog_manager.cc:5484] T 5c23236383d646d29f39a5c18938fd24 P 29f84c922f9f47f8bd7c360af478d359 reported cstate change: term changed from 0 to 1, leader changed from <none> to 29f84c922f9f47f8bd7c360af478d359 (127.9.176.193). New cstate: current_term: 1 leader_uuid: "29f84c922f9f47f8bd7c360af478d359" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29f84c922f9f47f8bd7c360af478d359" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 42839 } health_report { overall_health: HEALTHY } } }
I20240502 01:55:02.292852 23744 raft_consensus.cc:504] T 72b01a904eab46e7865356ab5358acb6 P 22b61cbb0f2148db8f62df6a32215c8d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b61cbb0f2148db8f62df6a32215c8d" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 35853 } }
I20240502 01:55:02.293073 23744 leader_election.cc:302] T 72b01a904eab46e7865356ab5358acb6 P 22b61cbb0f2148db8f62df6a32215c8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22b61cbb0f2148db8f62df6a32215c8d; no voters: 
I20240502 01:55:02.293267 23744 leader_election.cc:288] T 72b01a904eab46e7865356ab5358acb6 P 22b61cbb0f2148db8f62df6a32215c8d [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240502 01:55:02.293375 23747 raft_consensus.cc:2781] T 72b01a904eab46e7865356ab5358acb6 P 22b61cbb0f2148db8f62df6a32215c8d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:55:02.294970 23744 ts_tablet_manager.cc:1398] T 72b01a904eab46e7865356ab5358acb6 P 22b61cbb0f2148db8f62df6a32215c8d: Time spent starting tablet: real 0.009s	user 0.001s	sys 0.004s
I20240502 01:55:02.295073 23747 raft_consensus.cc:686] T 72b01a904eab46e7865356ab5358acb6 P 22b61cbb0f2148db8f62df6a32215c8d [term 1 LEADER]: Becoming Leader. State: Replica: 22b61cbb0f2148db8f62df6a32215c8d, State: Running, Role: LEADER
I20240502 01:55:02.295392 23747 consensus_queue.cc:237] T 72b01a904eab46e7865356ab5358acb6 P 22b61cbb0f2148db8f62df6a32215c8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b61cbb0f2148db8f62df6a32215c8d" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 35853 } }
I20240502 01:55:02.297576 23006 catalog_manager.cc:5484] T 0e2012468bb94a30a1d62cb58a98fc8c P 22b61cbb0f2148db8f62df6a32215c8d reported cstate change: term changed from 0 to 1, leader changed from <none> to 22b61cbb0f2148db8f62df6a32215c8d (127.9.176.194). New cstate: current_term: 1 leader_uuid: "22b61cbb0f2148db8f62df6a32215c8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b61cbb0f2148db8f62df6a32215c8d" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 35853 } health_report { overall_health: HEALTHY } } }
I20240502 01:55:02.302440 23006 catalog_manager.cc:5484] T 72b01a904eab46e7865356ab5358acb6 P 22b61cbb0f2148db8f62df6a32215c8d reported cstate change: term changed from 0 to 1, leader changed from <none> to 22b61cbb0f2148db8f62df6a32215c8d (127.9.176.194). New cstate: current_term: 1 leader_uuid: "22b61cbb0f2148db8f62df6a32215c8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b61cbb0f2148db8f62df6a32215c8d" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 35853 } health_report { overall_health: HEALTHY } } }
I20240502 01:55:03.765070  9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
/tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.195:34605
--local_ip_for_outbound_sockets=127.9.176.195
--tserver_master_addrs=127.9.176.254:44139
--webserver_port=40419
--webserver_interface=127.9.176.195
--builtin_ntp_servers=127.9.176.212:43017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 01:55:04.123797 23782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:55:04.124136 23782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:55:04.124281 23782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:55:04.132719 23782 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:55:04.132902 23782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:55:04.133067 23782 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.195
I20240502 01:55:04.142952 23782 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:43017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.195:34605
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.9.176.195
--webserver_port=40419
--tserver_master_addrs=127.9.176.254:44139
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.195
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:55:04.144385 23782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:55:04.146162 23782 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:55:04.158468 23787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:04.158874 23788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:04.159273 23790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:55:04.159827 23782 server_base.cc:1034] running on GCE node
I20240502 01:55:04.160487 23782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:55:04.161535 23782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:55:04.162752 23782 hybrid_clock.cc:648] HybridClock initialized: now 1714614904162698 us; error 56 us; skew 500 ppm
I20240502 01:55:04.163105 23782 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:55:04.165621 23782 webserver.cc:460] Webserver started at http://127.9.176.195:40419/ using document root <none> and password file <none>
I20240502 01:55:04.166443 23782 fs_manager.cc:359] Metadata directory not provided
I20240502 01:55:04.166585 23782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:55:04.172470 23782 fs_manager.cc:709] Time spent opening directory manager: real 0.004s	user 0.003s	sys 0.000s
I20240502 01:55:04.175824 23796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:55:04.177412 23782 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.001s
I20240502 01:55:04.177640 23782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "218f63e888ff4c6394af93014b59d65f"
format_stamp: "Formatted at 2024-05-02 01:54:53 on dist-test-slave-j3hh"
I20240502 01:55:04.178579 23782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:55:04.199162 23782 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:55:04.200323 23782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:55:04.200700 23782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:55:04.201913 23782 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:55:04.204396 23803 ts_tablet_manager.cc:522] Loading tablet metadata (0/1 complete)
I20240502 01:55:04.209769 23782 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240502 01:55:04.209897 23782 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.006s	user 0.001s	sys 0.000s
I20240502 01:55:04.210006 23782 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240502 01:55:04.213127 23782 ts_tablet_manager.cc:590] Registered 1 tablets
I20240502 01:55:04.213232 23782 ts_tablet_manager.cc:569] Time spent register tablets: real 0.003s	user 0.002s	sys 0.000s
I20240502 01:55:04.213606 23803 tablet_bootstrap.cc:492] T 6341db525c364256bc0450f381207ad3 P 218f63e888ff4c6394af93014b59d65f: Bootstrap starting.
I20240502 01:55:04.236189 23803 log.cc:824] T 6341db525c364256bc0450f381207ad3 P 218f63e888ff4c6394af93014b59d65f: Log is configured to *not* fsync() on all Append() calls
I20240502 01:55:04.241030 23803 tablet_bootstrap.cc:492] T 6341db525c364256bc0450f381207ad3 P 218f63e888ff4c6394af93014b59d65f: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:04.241726 23803 tablet_bootstrap.cc:492] T 6341db525c364256bc0450f381207ad3 P 218f63e888ff4c6394af93014b59d65f: Bootstrap complete.
I20240502 01:55:04.243010 23803 ts_tablet_manager.cc:1367] T 6341db525c364256bc0450f381207ad3 P 218f63e888ff4c6394af93014b59d65f: Time spent bootstrapping tablet: real 0.030s	user 0.015s	sys 0.010s
I20240502 01:55:04.255008 23803 raft_consensus.cc:348] T 6341db525c364256bc0450f381207ad3 P 218f63e888ff4c6394af93014b59d65f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "218f63e888ff4c6394af93014b59d65f" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 34605 } }
I20240502 01:55:04.256302 23803 raft_consensus.cc:729] T 6341db525c364256bc0450f381207ad3 P 218f63e888ff4c6394af93014b59d65f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 218f63e888ff4c6394af93014b59d65f, State: Initialized, Role: FOLLOWER
I20240502 01:55:04.257200 23803 consensus_queue.cc:260] T 6341db525c364256bc0450f381207ad3 P 218f63e888ff4c6394af93014b59d65f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "218f63e888ff4c6394af93014b59d65f" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 34605 } }
I20240502 01:55:04.257432 23803 raft_consensus.cc:388] T 6341db525c364256bc0450f381207ad3 P 218f63e888ff4c6394af93014b59d65f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:55:04.257591 23803 raft_consensus.cc:482] T 6341db525c364256bc0450f381207ad3 P 218f63e888ff4c6394af93014b59d65f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:55:04.257786 23803 raft_consensus.cc:3037] T 6341db525c364256bc0450f381207ad3 P 218f63e888ff4c6394af93014b59d65f [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:55:04.262475 23803 raft_consensus.cc:504] T 6341db525c364256bc0450f381207ad3 P 218f63e888ff4c6394af93014b59d65f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "218f63e888ff4c6394af93014b59d65f" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 34605 } }
I20240502 01:55:04.263082 23803 leader_election.cc:302] T 6341db525c364256bc0450f381207ad3 P 218f63e888ff4c6394af93014b59d65f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 218f63e888ff4c6394af93014b59d65f; no voters: 
I20240502 01:55:04.263903 23803 leader_election.cc:288] T 6341db525c364256bc0450f381207ad3 P 218f63e888ff4c6394af93014b59d65f [CANDIDATE]: Term 2 election: Requested vote from peers 
I20240502 01:55:04.264027 23912 raft_consensus.cc:2781] T 6341db525c364256bc0450f381207ad3 P 218f63e888ff4c6394af93014b59d65f [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:55:04.264084 23782 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.195:34605
I20240502 01:55:04.264179 23911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.195:34605 every 8 connection(s)
I20240502 01:55:04.264889 23912 raft_consensus.cc:686] T 6341db525c364256bc0450f381207ad3 P 218f63e888ff4c6394af93014b59d65f [term 2 LEADER]: Becoming Leader. State: Replica: 218f63e888ff4c6394af93014b59d65f, State: Running, Role: LEADER
I20240502 01:55:04.266147 23782 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 01:55:04.265844 23912 consensus_queue.cc:237] T 6341db525c364256bc0450f381207ad3 P 218f63e888ff4c6394af93014b59d65f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "218f63e888ff4c6394af93014b59d65f" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 34605 } }
I20240502 01:55:04.267689 23803 ts_tablet_manager.cc:1398] T 6341db525c364256bc0450f381207ad3 P 218f63e888ff4c6394af93014b59d65f: Time spent starting tablet: real 0.025s	user 0.013s	sys 0.015s
I20240502 01:55:04.272177  9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu as pid 23782
I20240502 01:55:04.273068  9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
/tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.196:45397
--local_ip_for_outbound_sockets=127.9.176.196
--tserver_master_addrs=127.9.176.254:44139
--webserver_port=44913
--webserver_interface=127.9.176.196
--builtin_ntp_servers=127.9.176.212:43017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:55:04.285420 23915 heartbeater.cc:347] Connected to a master server at 127.9.176.254:44139
I20240502 01:55:04.285773 23915 heartbeater.cc:464] Registering TS with master...
I20240502 01:55:04.286996 23915 heartbeater.cc:511] Master 127.9.176.254:44139 requested a full tablet report, sending...
I20240502 01:55:04.290295 23006 ts_manager.cc:194] Re-registered known tserver with Master: 218f63e888ff4c6394af93014b59d65f (127.9.176.195:34605)
I20240502 01:55:04.290891 23006 catalog_manager.cc:5484] T 6341db525c364256bc0450f381207ad3 P 218f63e888ff4c6394af93014b59d65f reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "218f63e888ff4c6394af93014b59d65f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "218f63e888ff4c6394af93014b59d65f" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 34605 } health_report { overall_health: HEALTHY } } }
I20240502 01:55:04.296813 23006 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.176.195:52283
I20240502 01:55:04.299978 23915 heartbeater.cc:503] Master 127.9.176.254:44139 was elected leader, sending a full tablet report...
W20240502 01:55:04.643035 23921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:55:04.643347 23921 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:55:04.643489 23921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:55:04.651208 23921 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:55:04.651350 23921 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:55:04.651527 23921 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.196
I20240502 01:55:04.660946 23921 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:43017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.196:45397
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.9.176.196
--webserver_port=44913
--tserver_master_addrs=127.9.176.254:44139
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.196
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:55:04.662341 23921 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:55:04.664213 23921 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:55:04.676821 23930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:04.677443 23931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:04.677842 23933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:55:04.678526 23921 server_base.cc:1034] running on GCE node
I20240502 01:55:04.679277 23921 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:55:04.680310 23921 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:55:04.681521 23921 hybrid_clock.cc:648] HybridClock initialized: now 1714614904681470 us; error 44 us; skew 500 ppm
I20240502 01:55:04.681869 23921 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:55:04.684386 23921 webserver.cc:460] Webserver started at http://127.9.176.196:44913/ using document root <none> and password file <none>
I20240502 01:55:04.685271 23921 fs_manager.cc:359] Metadata directory not provided
I20240502 01:55:04.685408 23921 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:55:04.691005 23921 fs_manager.cc:709] Time spent opening directory manager: real 0.004s	user 0.005s	sys 0.000s
I20240502 01:55:04.694120 23939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:55:04.695623 23921 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20240502 01:55:04.695822 23921 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "631566d8d4dc4a43a1a391ea74418418"
format_stamp: "Formatted at 2024-05-02 01:54:54 on dist-test-slave-j3hh"
I20240502 01:55:04.696731 23921 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:55:04.726267 23921 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:55:04.727146 23921 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:55:04.727432 23921 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:55:04.728575 23921 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:55:04.730250 23921 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:55:04.730331 23921 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:55:04.730465 23921 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:55:04.730517 23921 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:55:04.770364 23921 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.196:45397
I20240502 01:55:04.770416 24052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.196:45397 every 8 connection(s)
I20240502 01:55:04.771981 23921 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 01:55:04.773308  9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu as pid 23921
I20240502 01:55:04.788444 24053 heartbeater.cc:347] Connected to a master server at 127.9.176.254:44139
I20240502 01:55:04.788789 24053 heartbeater.cc:464] Registering TS with master...
I20240502 01:55:04.789642 24053 heartbeater.cc:511] Master 127.9.176.254:44139 requested a full tablet report, sending...
I20240502 01:55:04.791302 23006 ts_manager.cc:194] Re-registered known tserver with Master: 631566d8d4dc4a43a1a391ea74418418 (127.9.176.196:45397)
I20240502 01:55:04.792640 23006 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.176.196:49615
I20240502 01:55:05.297127 23259 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:55:05.302819 23128 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:55:05.315275 23845 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:55:05.323087 23988 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:05.498798 23845 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:55:05.504411 23259 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:55:05.506201 23128 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:55:05.516183 23988 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:05.723001 23988 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:05.729960 23845 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:55:05.730000 23259 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:55:05.730170 23128 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:55:05.795743 24053 heartbeater.cc:503] Master 127.9.176.254:44139 was elected leader, sending a full tablet report...
I20240502 01:55:05.829249 23279 consensus_queue.cc:237] T 6213c1863b2048c1a281634dd2693c3e P 22b61cbb0f2148db8f62df6a32215c8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "22b61cbb0f2148db8f62df6a32215c8d" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 35853 } attrs { replace: true } } peers { permanent_uuid: "631566d8d4dc4a43a1a391ea74418418" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 45397 } attrs { promote: true } }
I20240502 01:55:05.832149 24170 raft_consensus.cc:2932] T 6213c1863b2048c1a281634dd2693c3e P 22b61cbb0f2148db8f62df6a32215c8d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 631566d8d4dc4a43a1a391ea74418418 (127.9.176.196) added. New config: { opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "22b61cbb0f2148db8f62df6a32215c8d" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 35853 } attrs { replace: true } } peers { permanent_uuid: "631566d8d4dc4a43a1a391ea74418418" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 45397 } attrs { promote: true } } }
I20240502 01:55:05.838147 23005 catalog_manager.cc:5484] T 6213c1863b2048c1a281634dd2693c3e P 22b61cbb0f2148db8f62df6a32215c8d reported cstate change: config changed from index -1 to 2, NON_VOTER 631566d8d4dc4a43a1a391ea74418418 (127.9.176.196) added. New cstate: current_term: 1 leader_uuid: "22b61cbb0f2148db8f62df6a32215c8d" committed_config { opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "22b61cbb0f2148db8f62df6a32215c8d" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 35853 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "631566d8d4dc4a43a1a391ea74418418" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 45397 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 01:55:05.842679 23213 consensus_peers.cc:485] T 6213c1863b2048c1a281634dd2693c3e P 22b61cbb0f2148db8f62df6a32215c8d -> Peer 631566d8d4dc4a43a1a391ea74418418 (127.9.176.196:45397): Couldn't send request to peer 631566d8d4dc4a43a1a391ea74418418. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6213c1863b2048c1a281634dd2693c3e. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:05.853703 23148 consensus_queue.cc:237] T 49eddfad6f6e4247a83e82964e8a9470 P 29f84c922f9f47f8bd7c360af478d359 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "29f84c922f9f47f8bd7c360af478d359" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 42839 } attrs { replace: true } } peers { permanent_uuid: "218f63e888ff4c6394af93014b59d65f" member_type: NON_VOTER last_known_addr { host: "127.9.176.195" port: 34605 } attrs { promote: true } }
I20240502 01:55:05.856626 24183 raft_consensus.cc:2932] T 49eddfad6f6e4247a83e82964e8a9470 P 29f84c922f9f47f8bd7c360af478d359 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index -1 to 3, NON_VOTER 218f63e888ff4c6394af93014b59d65f (127.9.176.195) added. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "29f84c922f9f47f8bd7c360af478d359" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 42839 } attrs { replace: true } } peers { permanent_uuid: "218f63e888ff4c6394af93014b59d65f" member_type: NON_VOTER last_known_addr { host: "127.9.176.195" port: 34605 } attrs { promote: true } } }
I20240502 01:55:05.863006 23005 catalog_manager.cc:5484] T 49eddfad6f6e4247a83e82964e8a9470 P 29f84c922f9f47f8bd7c360af478d359 reported cstate change: config changed from index -1 to 3, NON_VOTER 218f63e888ff4c6394af93014b59d65f (127.9.176.195) added. New cstate: current_term: 1 leader_uuid: "29f84c922f9f47f8bd7c360af478d359" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "29f84c922f9f47f8bd7c360af478d359" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 42839 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "218f63e888ff4c6394af93014b59d65f" member_type: NON_VOTER last_known_addr { host: "127.9.176.195" port: 34605 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 01:55:05.866119 23083 consensus_peers.cc:485] T 49eddfad6f6e4247a83e82964e8a9470 P 29f84c922f9f47f8bd7c360af478d359 -> Peer 218f63e888ff4c6394af93014b59d65f (127.9.176.195:34605): Couldn't send request to peer 218f63e888ff4c6394af93014b59d65f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 49eddfad6f6e4247a83e82964e8a9470. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:05.876854 23279 consensus_queue.cc:237] T 72b01a904eab46e7865356ab5358acb6 P 22b61cbb0f2148db8f62df6a32215c8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "22b61cbb0f2148db8f62df6a32215c8d" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 35853 } attrs { replace: true } } peers { permanent_uuid: "631566d8d4dc4a43a1a391ea74418418" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 45397 } attrs { promote: true } }
I20240502 01:55:05.878598 24169 raft_consensus.cc:2932] T 72b01a904eab46e7865356ab5358acb6 P 22b61cbb0f2148db8f62df6a32215c8d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index -1 to 3, NON_VOTER 631566d8d4dc4a43a1a391ea74418418 (127.9.176.196) added. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "22b61cbb0f2148db8f62df6a32215c8d" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 35853 } attrs { replace: true } } peers { permanent_uuid: "631566d8d4dc4a43a1a391ea74418418" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 45397 } attrs { promote: true } } }
W20240502 01:55:05.879086 23213 consensus_peers.cc:485] T 72b01a904eab46e7865356ab5358acb6 P 22b61cbb0f2148db8f62df6a32215c8d -> Peer 631566d8d4dc4a43a1a391ea74418418 (127.9.176.196:45397): Couldn't send request to peer 631566d8d4dc4a43a1a391ea74418418. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 72b01a904eab46e7865356ab5358acb6. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:05.883584 23005 catalog_manager.cc:5484] T 72b01a904eab46e7865356ab5358acb6 P 22b61cbb0f2148db8f62df6a32215c8d reported cstate change: config changed from index -1 to 3, NON_VOTER 631566d8d4dc4a43a1a391ea74418418 (127.9.176.196) added. New cstate: current_term: 1 leader_uuid: "22b61cbb0f2148db8f62df6a32215c8d" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "22b61cbb0f2148db8f62df6a32215c8d" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 35853 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "631566d8d4dc4a43a1a391ea74418418" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 45397 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:55:05.895836 23148 consensus_queue.cc:237] T 1f4cf78efc5f479c875536136d17fdb0 P 29f84c922f9f47f8bd7c360af478d359 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "29f84c922f9f47f8bd7c360af478d359" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 42839 } attrs { replace: true } } peers { permanent_uuid: "218f63e888ff4c6394af93014b59d65f" member_type: NON_VOTER last_known_addr { host: "127.9.176.195" port: 34605 } attrs { promote: true } }
I20240502 01:55:05.897477 24183 raft_consensus.cc:2932] T 1f4cf78efc5f479c875536136d17fdb0 P 29f84c922f9f47f8bd7c360af478d359 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 218f63e888ff4c6394af93014b59d65f (127.9.176.195) added. New config: { opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "29f84c922f9f47f8bd7c360af478d359" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 42839 } attrs { replace: true } } peers { permanent_uuid: "218f63e888ff4c6394af93014b59d65f" member_type: NON_VOTER last_known_addr { host: "127.9.176.195" port: 34605 } attrs { promote: true } } }
W20240502 01:55:05.897970 23083 consensus_peers.cc:485] T 1f4cf78efc5f479c875536136d17fdb0 P 29f84c922f9f47f8bd7c360af478d359 -> Peer 218f63e888ff4c6394af93014b59d65f (127.9.176.195:34605): Couldn't send request to peer 218f63e888ff4c6394af93014b59d65f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1f4cf78efc5f479c875536136d17fdb0. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:05.902494 23005 catalog_manager.cc:5484] T 1f4cf78efc5f479c875536136d17fdb0 P 29f84c922f9f47f8bd7c360af478d359 reported cstate change: config changed from index -1 to 2, NON_VOTER 218f63e888ff4c6394af93014b59d65f (127.9.176.195) added. New cstate: current_term: 1 leader_uuid: "29f84c922f9f47f8bd7c360af478d359" committed_config { opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "29f84c922f9f47f8bd7c360af478d359" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 42839 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "218f63e888ff4c6394af93014b59d65f" member_type: NON_VOTER last_known_addr { host: "127.9.176.195" port: 34605 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:55:05.914083 23148 consensus_queue.cc:237] T 209d6a3dc32f43b1b4d536e02bd0ec73 P 29f84c922f9f47f8bd7c360af478d359 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "29f84c922f9f47f8bd7c360af478d359" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 42839 } attrs { replace: true } } peers { permanent_uuid: "631566d8d4dc4a43a1a391ea74418418" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 45397 } attrs { promote: true } }
I20240502 01:55:05.916199 24182 raft_consensus.cc:2932] T 209d6a3dc32f43b1b4d536e02bd0ec73 P 29f84c922f9f47f8bd7c360af478d359 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 631566d8d4dc4a43a1a391ea74418418 (127.9.176.196) added. New config: { opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "29f84c922f9f47f8bd7c360af478d359" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 42839 } attrs { replace: true } } peers { permanent_uuid: "631566d8d4dc4a43a1a391ea74418418" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 45397 } attrs { promote: true } } }
I20240502 01:55:05.921710 23005 catalog_manager.cc:5484] T 209d6a3dc32f43b1b4d536e02bd0ec73 P 29f84c922f9f47f8bd7c360af478d359 reported cstate change: config changed from index -1 to 2, NON_VOTER 631566d8d4dc4a43a1a391ea74418418 (127.9.176.196) added. New cstate: current_term: 1 leader_uuid: "29f84c922f9f47f8bd7c360af478d359" committed_config { opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "29f84c922f9f47f8bd7c360af478d359" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 42839 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "631566d8d4dc4a43a1a391ea74418418" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 45397 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 01:55:05.923674 23082 consensus_peers.cc:485] T 209d6a3dc32f43b1b4d536e02bd0ec73 P 29f84c922f9f47f8bd7c360af478d359 -> Peer 631566d8d4dc4a43a1a391ea74418418 (127.9.176.196:45397): Couldn't send request to peer 631566d8d4dc4a43a1a391ea74418418. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 209d6a3dc32f43b1b4d536e02bd0ec73. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:06.266988 24254 ts_tablet_manager.cc:902] T 49eddfad6f6e4247a83e82964e8a9470 P 218f63e888ff4c6394af93014b59d65f: Initiating tablet copy from peer 29f84c922f9f47f8bd7c360af478d359 (127.9.176.193:42839)
I20240502 01:55:06.269110 24254 tablet_copy_client.cc:308] T 49eddfad6f6e4247a83e82964e8a9470 P 218f63e888ff4c6394af93014b59d65f: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.193:42839
I20240502 01:55:06.276218 23168 tablet_copy_service.cc:140] P 29f84c922f9f47f8bd7c360af478d359: Received BeginTabletCopySession request for tablet 49eddfad6f6e4247a83e82964e8a9470 from peer 218f63e888ff4c6394af93014b59d65f ({username='slave'} at 127.9.176.195:37419)
I20240502 01:55:06.276499 23168 tablet_copy_service.cc:161] P 29f84c922f9f47f8bd7c360af478d359: Beginning new tablet copy session on tablet 49eddfad6f6e4247a83e82964e8a9470 from peer 218f63e888ff4c6394af93014b59d65f at {username='slave'} at 127.9.176.195:37419: session id = 218f63e888ff4c6394af93014b59d65f-49eddfad6f6e4247a83e82964e8a9470
I20240502 01:55:06.279404 23168 tablet_copy_source_session.cc:205] T 49eddfad6f6e4247a83e82964e8a9470 P 29f84c922f9f47f8bd7c360af478d359: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:06.282505 24254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49eddfad6f6e4247a83e82964e8a9470. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:06.288440 24260 ts_tablet_manager.cc:902] T 72b01a904eab46e7865356ab5358acb6 P 631566d8d4dc4a43a1a391ea74418418: Initiating tablet copy from peer 22b61cbb0f2148db8f62df6a32215c8d (127.9.176.194:35853)
I20240502 01:55:06.289089 24254 tablet_copy_client.cc:783] T 49eddfad6f6e4247a83e82964e8a9470 P 218f63e888ff4c6394af93014b59d65f: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:06.289552 24254 tablet_copy_client.cc:647] T 49eddfad6f6e4247a83e82964e8a9470 P 218f63e888ff4c6394af93014b59d65f: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:06.290508 24260 tablet_copy_client.cc:308] T 72b01a904eab46e7865356ab5358acb6 P 631566d8d4dc4a43a1a391ea74418418: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.194:35853
I20240502 01:55:06.292470 24254 tablet_copy_client.cc:523] T 49eddfad6f6e4247a83e82964e8a9470 P 218f63e888ff4c6394af93014b59d65f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:06.295389 24254 tablet_bootstrap.cc:492] T 49eddfad6f6e4247a83e82964e8a9470 P 218f63e888ff4c6394af93014b59d65f: Bootstrap starting.
I20240502 01:55:06.298020 23299 tablet_copy_service.cc:140] P 22b61cbb0f2148db8f62df6a32215c8d: Received BeginTabletCopySession request for tablet 72b01a904eab46e7865356ab5358acb6 from peer 631566d8d4dc4a43a1a391ea74418418 ({username='slave'} at 127.9.176.196:40989)
I20240502 01:55:06.298269 23299 tablet_copy_service.cc:161] P 22b61cbb0f2148db8f62df6a32215c8d: Beginning new tablet copy session on tablet 72b01a904eab46e7865356ab5358acb6 from peer 631566d8d4dc4a43a1a391ea74418418 at {username='slave'} at 127.9.176.196:40989: session id = 631566d8d4dc4a43a1a391ea74418418-72b01a904eab46e7865356ab5358acb6
I20240502 01:55:06.301435 23299 tablet_copy_source_session.cc:205] T 72b01a904eab46e7865356ab5358acb6 P 22b61cbb0f2148db8f62df6a32215c8d: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:06.306968 24260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72b01a904eab46e7865356ab5358acb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:06.307786 24254 tablet_bootstrap.cc:492] T 49eddfad6f6e4247a83e82964e8a9470 P 218f63e888ff4c6394af93014b59d65f: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:06.308207 24254 tablet_bootstrap.cc:492] T 49eddfad6f6e4247a83e82964e8a9470 P 218f63e888ff4c6394af93014b59d65f: Bootstrap complete.
I20240502 01:55:06.308467 24254 ts_tablet_manager.cc:1367] T 49eddfad6f6e4247a83e82964e8a9470 P 218f63e888ff4c6394af93014b59d65f: Time spent bootstrapping tablet: real 0.013s	user 0.011s	sys 0.004s
I20240502 01:55:06.309309 24254 raft_consensus.cc:348] T 49eddfad6f6e4247a83e82964e8a9470 P 218f63e888ff4c6394af93014b59d65f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "29f84c922f9f47f8bd7c360af478d359" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 42839 } attrs { replace: true } } peers { permanent_uuid: "218f63e888ff4c6394af93014b59d65f" member_type: NON_VOTER last_known_addr { host: "127.9.176.195" port: 34605 } attrs { promote: true } }
I20240502 01:55:06.309510 24254 raft_consensus.cc:729] T 49eddfad6f6e4247a83e82964e8a9470 P 218f63e888ff4c6394af93014b59d65f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 218f63e888ff4c6394af93014b59d65f, State: Initialized, Role: LEARNER
I20240502 01:55:06.309767 24254 consensus_queue.cc:260] T 49eddfad6f6e4247a83e82964e8a9470 P 218f63e888ff4c6394af93014b59d65f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "29f84c922f9f47f8bd7c360af478d359" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 42839 } attrs { replace: true } } peers { permanent_uuid: "218f63e888ff4c6394af93014b59d65f" member_type: NON_VOTER last_known_addr { host: "127.9.176.195" port: 34605 } attrs { promote: true } }
I20240502 01:55:06.310655 24254 ts_tablet_manager.cc:1398] T 49eddfad6f6e4247a83e82964e8a9470 P 218f63e888ff4c6394af93014b59d65f: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.001s
I20240502 01:55:06.311775 23168 tablet_copy_service.cc:342] P 29f84c922f9f47f8bd7c360af478d359: Request end of tablet copy session 218f63e888ff4c6394af93014b59d65f-49eddfad6f6e4247a83e82964e8a9470 received from {username='slave'} at 127.9.176.195:37419
I20240502 01:55:06.311998 23168 tablet_copy_service.cc:433] P 29f84c922f9f47f8bd7c360af478d359: ending tablet copy session 218f63e888ff4c6394af93014b59d65f-49eddfad6f6e4247a83e82964e8a9470 on tablet 49eddfad6f6e4247a83e82964e8a9470 with peer 218f63e888ff4c6394af93014b59d65f
I20240502 01:55:06.320350 24260 tablet_copy_client.cc:783] T 72b01a904eab46e7865356ab5358acb6 P 631566d8d4dc4a43a1a391ea74418418: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:06.320753 24260 tablet_copy_client.cc:647] T 72b01a904eab46e7865356ab5358acb6 P 631566d8d4dc4a43a1a391ea74418418: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:06.323256 24260 tablet_copy_client.cc:523] T 72b01a904eab46e7865356ab5358acb6 P 631566d8d4dc4a43a1a391ea74418418: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:06.326651 24260 tablet_bootstrap.cc:492] T 72b01a904eab46e7865356ab5358acb6 P 631566d8d4dc4a43a1a391ea74418418: Bootstrap starting.
I20240502 01:55:06.334595 24260 log.cc:824] T 72b01a904eab46e7865356ab5358acb6 P 631566d8d4dc4a43a1a391ea74418418: Log is configured to *not* fsync() on all Append() calls
I20240502 01:55:06.344204 24260 tablet_bootstrap.cc:492] T 72b01a904eab46e7865356ab5358acb6 P 631566d8d4dc4a43a1a391ea74418418: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:06.344687 24260 tablet_bootstrap.cc:492] T 72b01a904eab46e7865356ab5358acb6 P 631566d8d4dc4a43a1a391ea74418418: Bootstrap complete.
I20240502 01:55:06.345065 24260 ts_tablet_manager.cc:1367] T 72b01a904eab46e7865356ab5358acb6 P 631566d8d4dc4a43a1a391ea74418418: Time spent bootstrapping tablet: real 0.019s	user 0.011s	sys 0.009s
I20240502 01:55:06.349857 24260 raft_consensus.cc:348] T 72b01a904eab46e7865356ab5358acb6 P 631566d8d4dc4a43a1a391ea74418418 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "22b61cbb0f2148db8f62df6a32215c8d" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 35853 } attrs { replace: true } } peers { permanent_uuid: "631566d8d4dc4a43a1a391ea74418418" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 45397 } attrs { promote: true } }
I20240502 01:55:06.350251 24260 raft_consensus.cc:729] T 72b01a904eab46e7865356ab5358acb6 P 631566d8d4dc4a43a1a391ea74418418 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 631566d8d4dc4a43a1a391ea74418418, State: Initialized, Role: LEARNER
I20240502 01:55:06.350695 24260 consensus_queue.cc:260] T 72b01a904eab46e7865356ab5358acb6 P 631566d8d4dc4a43a1a391ea74418418 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "22b61cbb0f2148db8f62df6a32215c8d" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 35853 } attrs { replace: true } } peers { permanent_uuid: "631566d8d4dc4a43a1a391ea74418418" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 45397 } attrs { promote: true } }
I20240502 01:55:06.352165 24260 ts_tablet_manager.cc:1398] T 72b01a904eab46e7865356ab5358acb6 P 631566d8d4dc4a43a1a391ea74418418: Time spent starting tablet: real 0.007s	user 0.003s	sys 0.005s
I20240502 01:55:06.353209 23299 tablet_copy_service.cc:342] P 22b61cbb0f2148db8f62df6a32215c8d: Request end of tablet copy session 631566d8d4dc4a43a1a391ea74418418-72b01a904eab46e7865356ab5358acb6 received from {username='slave'} at 127.9.176.196:40989
I20240502 01:55:06.353392 23299 tablet_copy_service.cc:433] P 22b61cbb0f2148db8f62df6a32215c8d: ending tablet copy session 631566d8d4dc4a43a1a391ea74418418-72b01a904eab46e7865356ab5358acb6 on tablet 72b01a904eab46e7865356ab5358acb6 with peer 631566d8d4dc4a43a1a391ea74418418
I20240502 01:55:06.416918 24260 ts_tablet_manager.cc:902] T 6213c1863b2048c1a281634dd2693c3e P 631566d8d4dc4a43a1a391ea74418418: Initiating tablet copy from peer 22b61cbb0f2148db8f62df6a32215c8d (127.9.176.194:35853)
I20240502 01:55:06.418104 24260 tablet_copy_client.cc:308] T 6213c1863b2048c1a281634dd2693c3e P 631566d8d4dc4a43a1a391ea74418418: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.194:35853
I20240502 01:55:06.418905 23299 tablet_copy_service.cc:140] P 22b61cbb0f2148db8f62df6a32215c8d: Received BeginTabletCopySession request for tablet 6213c1863b2048c1a281634dd2693c3e from peer 631566d8d4dc4a43a1a391ea74418418 ({username='slave'} at 127.9.176.196:40989)
I20240502 01:55:06.419095 23299 tablet_copy_service.cc:161] P 22b61cbb0f2148db8f62df6a32215c8d: Beginning new tablet copy session on tablet 6213c1863b2048c1a281634dd2693c3e from peer 631566d8d4dc4a43a1a391ea74418418 at {username='slave'} at 127.9.176.196:40989: session id = 631566d8d4dc4a43a1a391ea74418418-6213c1863b2048c1a281634dd2693c3e
I20240502 01:55:06.421160 23299 tablet_copy_source_session.cc:205] T 6213c1863b2048c1a281634dd2693c3e P 22b61cbb0f2148db8f62df6a32215c8d: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:06.422463 24260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6213c1863b2048c1a281634dd2693c3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:06.427160 24260 tablet_copy_client.cc:783] T 6213c1863b2048c1a281634dd2693c3e P 631566d8d4dc4a43a1a391ea74418418: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:06.427410 24260 tablet_copy_client.cc:647] T 6213c1863b2048c1a281634dd2693c3e P 631566d8d4dc4a43a1a391ea74418418: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:06.428697 24254 ts_tablet_manager.cc:902] T 1f4cf78efc5f479c875536136d17fdb0 P 218f63e888ff4c6394af93014b59d65f: Initiating tablet copy from peer 29f84c922f9f47f8bd7c360af478d359 (127.9.176.193:42839)
I20240502 01:55:06.429329 24260 tablet_copy_client.cc:523] T 6213c1863b2048c1a281634dd2693c3e P 631566d8d4dc4a43a1a391ea74418418: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:06.430032 24254 tablet_copy_client.cc:308] T 1f4cf78efc5f479c875536136d17fdb0 P 218f63e888ff4c6394af93014b59d65f: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.193:42839
I20240502 01:55:06.430768 23168 tablet_copy_service.cc:140] P 29f84c922f9f47f8bd7c360af478d359: Received BeginTabletCopySession request for tablet 1f4cf78efc5f479c875536136d17fdb0 from peer 218f63e888ff4c6394af93014b59d65f ({username='slave'} at 127.9.176.195:37419)
I20240502 01:55:06.430944 23168 tablet_copy_service.cc:161] P 29f84c922f9f47f8bd7c360af478d359: Beginning new tablet copy session on tablet 1f4cf78efc5f479c875536136d17fdb0 from peer 218f63e888ff4c6394af93014b59d65f at {username='slave'} at 127.9.176.195:37419: session id = 218f63e888ff4c6394af93014b59d65f-1f4cf78efc5f479c875536136d17fdb0
I20240502 01:55:06.432258 24260 tablet_bootstrap.cc:492] T 6213c1863b2048c1a281634dd2693c3e P 631566d8d4dc4a43a1a391ea74418418: Bootstrap starting.
I20240502 01:55:06.433208 23168 tablet_copy_source_session.cc:205] T 1f4cf78efc5f479c875536136d17fdb0 P 29f84c922f9f47f8bd7c360af478d359: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:06.434463 24254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f4cf78efc5f479c875536136d17fdb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:06.437639 24260 tablet_bootstrap.cc:492] T 6213c1863b2048c1a281634dd2693c3e P 631566d8d4dc4a43a1a391ea74418418: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:06.438017 24260 tablet_bootstrap.cc:492] T 6213c1863b2048c1a281634dd2693c3e P 631566d8d4dc4a43a1a391ea74418418: Bootstrap complete.
I20240502 01:55:06.438269 24260 ts_tablet_manager.cc:1367] T 6213c1863b2048c1a281634dd2693c3e P 631566d8d4dc4a43a1a391ea74418418: Time spent bootstrapping tablet: real 0.006s	user 0.001s	sys 0.003s
I20240502 01:55:06.439116 24260 raft_consensus.cc:348] T 6213c1863b2048c1a281634dd2693c3e P 631566d8d4dc4a43a1a391ea74418418 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "22b61cbb0f2148db8f62df6a32215c8d" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 35853 } attrs { replace: true } } peers { permanent_uuid: "631566d8d4dc4a43a1a391ea74418418" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 45397 } attrs { promote: true } }
I20240502 01:55:06.439314 24260 raft_consensus.cc:729] T 6213c1863b2048c1a281634dd2693c3e P 631566d8d4dc4a43a1a391ea74418418 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 631566d8d4dc4a43a1a391ea74418418, State: Initialized, Role: LEARNER
I20240502 01:55:06.439503 24260 consensus_queue.cc:260] T 6213c1863b2048c1a281634dd2693c3e P 631566d8d4dc4a43a1a391ea74418418 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "22b61cbb0f2148db8f62df6a32215c8d" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 35853 } attrs { replace: true } } peers { permanent_uuid: "631566d8d4dc4a43a1a391ea74418418" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 45397 } attrs { promote: true } }
I20240502 01:55:06.440102 24260 ts_tablet_manager.cc:1398] T 6213c1863b2048c1a281634dd2693c3e P 631566d8d4dc4a43a1a391ea74418418: Time spent starting tablet: real 0.002s	user 0.004s	sys 0.000s
I20240502 01:55:06.441023 23299 tablet_copy_service.cc:342] P 22b61cbb0f2148db8f62df6a32215c8d: Request end of tablet copy session 631566d8d4dc4a43a1a391ea74418418-6213c1863b2048c1a281634dd2693c3e received from {username='slave'} at 127.9.176.196:40989
I20240502 01:55:06.441107 24254 tablet_copy_client.cc:783] T 1f4cf78efc5f479c875536136d17fdb0 P 218f63e888ff4c6394af93014b59d65f: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:06.441152 23299 tablet_copy_service.cc:433] P 22b61cbb0f2148db8f62df6a32215c8d: ending tablet copy session 631566d8d4dc4a43a1a391ea74418418-6213c1863b2048c1a281634dd2693c3e on tablet 6213c1863b2048c1a281634dd2693c3e with peer 631566d8d4dc4a43a1a391ea74418418
I20240502 01:55:06.441429 24254 tablet_copy_client.cc:647] T 1f4cf78efc5f479c875536136d17fdb0 P 218f63e888ff4c6394af93014b59d65f: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:06.443500 24254 tablet_copy_client.cc:523] T 1f4cf78efc5f479c875536136d17fdb0 P 218f63e888ff4c6394af93014b59d65f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:06.446659 24254 tablet_bootstrap.cc:492] T 1f4cf78efc5f479c875536136d17fdb0 P 218f63e888ff4c6394af93014b59d65f: Bootstrap starting.
I20240502 01:55:06.447378 24260 ts_tablet_manager.cc:902] T 209d6a3dc32f43b1b4d536e02bd0ec73 P 631566d8d4dc4a43a1a391ea74418418: Initiating tablet copy from peer 29f84c922f9f47f8bd7c360af478d359 (127.9.176.193:42839)
I20240502 01:55:06.451944 24260 tablet_copy_client.cc:308] T 209d6a3dc32f43b1b4d536e02bd0ec73 P 631566d8d4dc4a43a1a391ea74418418: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.193:42839
I20240502 01:55:06.453822 24254 tablet_bootstrap.cc:492] T 1f4cf78efc5f479c875536136d17fdb0 P 218f63e888ff4c6394af93014b59d65f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:06.454406 24254 tablet_bootstrap.cc:492] T 1f4cf78efc5f479c875536136d17fdb0 P 218f63e888ff4c6394af93014b59d65f: Bootstrap complete.
I20240502 01:55:06.454727 24254 ts_tablet_manager.cc:1367] T 1f4cf78efc5f479c875536136d17fdb0 P 218f63e888ff4c6394af93014b59d65f: Time spent bootstrapping tablet: real 0.008s	user 0.007s	sys 0.000s
I20240502 01:55:06.455816 24254 raft_consensus.cc:348] T 1f4cf78efc5f479c875536136d17fdb0 P 218f63e888ff4c6394af93014b59d65f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "29f84c922f9f47f8bd7c360af478d359" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 42839 } attrs { replace: true } } peers { permanent_uuid: "218f63e888ff4c6394af93014b59d65f" member_type: NON_VOTER last_known_addr { host: "127.9.176.195" port: 34605 } attrs { promote: true } }
I20240502 01:55:06.456167 24254 raft_consensus.cc:729] T 1f4cf78efc5f479c875536136d17fdb0 P 218f63e888ff4c6394af93014b59d65f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 218f63e888ff4c6394af93014b59d65f, State: Initialized, Role: LEARNER
I20240502 01:55:06.456439 24254 consensus_queue.cc:260] T 1f4cf78efc5f479c875536136d17fdb0 P 218f63e888ff4c6394af93014b59d65f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "29f84c922f9f47f8bd7c360af478d359" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 42839 } attrs { replace: true } } peers { permanent_uuid: "218f63e888ff4c6394af93014b59d65f" member_type: NON_VOTER last_known_addr { host: "127.9.176.195" port: 34605 } attrs { promote: true } }
I20240502 01:55:06.457015 24254 ts_tablet_manager.cc:1398] T 1f4cf78efc5f479c875536136d17fdb0 P 218f63e888ff4c6394af93014b59d65f: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.002s
I20240502 01:55:06.457958 23168 tablet_copy_service.cc:342] P 29f84c922f9f47f8bd7c360af478d359: Request end of tablet copy session 218f63e888ff4c6394af93014b59d65f-1f4cf78efc5f479c875536136d17fdb0 received from {username='slave'} at 127.9.176.195:37419
I20240502 01:55:06.458101 23168 tablet_copy_service.cc:433] P 29f84c922f9f47f8bd7c360af478d359: ending tablet copy session 218f63e888ff4c6394af93014b59d65f-1f4cf78efc5f479c875536136d17fdb0 on tablet 1f4cf78efc5f479c875536136d17fdb0 with peer 218f63e888ff4c6394af93014b59d65f
I20240502 01:55:06.462107 23168 tablet_copy_service.cc:140] P 29f84c922f9f47f8bd7c360af478d359: Received BeginTabletCopySession request for tablet 209d6a3dc32f43b1b4d536e02bd0ec73 from peer 631566d8d4dc4a43a1a391ea74418418 ({username='slave'} at 127.9.176.196:36529)
I20240502 01:55:06.462272 23168 tablet_copy_service.cc:161] P 29f84c922f9f47f8bd7c360af478d359: Beginning new tablet copy session on tablet 209d6a3dc32f43b1b4d536e02bd0ec73 from peer 631566d8d4dc4a43a1a391ea74418418 at {username='slave'} at 127.9.176.196:36529: session id = 631566d8d4dc4a43a1a391ea74418418-209d6a3dc32f43b1b4d536e02bd0ec73
I20240502 01:55:06.465224 23168 tablet_copy_source_session.cc:205] T 209d6a3dc32f43b1b4d536e02bd0ec73 P 29f84c922f9f47f8bd7c360af478d359: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:06.466560 24260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 209d6a3dc32f43b1b4d536e02bd0ec73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:06.471972 24260 tablet_copy_client.cc:783] T 209d6a3dc32f43b1b4d536e02bd0ec73 P 631566d8d4dc4a43a1a391ea74418418: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:06.472201 24260 tablet_copy_client.cc:647] T 209d6a3dc32f43b1b4d536e02bd0ec73 P 631566d8d4dc4a43a1a391ea74418418: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:06.473841 24260 tablet_copy_client.cc:523] T 209d6a3dc32f43b1b4d536e02bd0ec73 P 631566d8d4dc4a43a1a391ea74418418: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:06.476426 24260 tablet_bootstrap.cc:492] T 209d6a3dc32f43b1b4d536e02bd0ec73 P 631566d8d4dc4a43a1a391ea74418418: Bootstrap starting.
I20240502 01:55:06.481858 24260 tablet_bootstrap.cc:492] T 209d6a3dc32f43b1b4d536e02bd0ec73 P 631566d8d4dc4a43a1a391ea74418418: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:06.482314 24260 tablet_bootstrap.cc:492] T 209d6a3dc32f43b1b4d536e02bd0ec73 P 631566d8d4dc4a43a1a391ea74418418: Bootstrap complete.
I20240502 01:55:06.482614 24260 ts_tablet_manager.cc:1367] T 209d6a3dc32f43b1b4d536e02bd0ec73 P 631566d8d4dc4a43a1a391ea74418418: Time spent bootstrapping tablet: real 0.006s	user 0.004s	sys 0.000s
I20240502 01:55:06.483297 24260 raft_consensus.cc:348] T 209d6a3dc32f43b1b4d536e02bd0ec73 P 631566d8d4dc4a43a1a391ea74418418 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "29f84c922f9f47f8bd7c360af478d359" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 42839 } attrs { replace: true } } peers { permanent_uuid: "631566d8d4dc4a43a1a391ea74418418" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 45397 } attrs { promote: true } }
I20240502 01:55:06.483511 24260 raft_consensus.cc:729] T 209d6a3dc32f43b1b4d536e02bd0ec73 P 631566d8d4dc4a43a1a391ea74418418 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 631566d8d4dc4a43a1a391ea74418418, State: Initialized, Role: LEARNER
I20240502 01:55:06.483757 24260 consensus_queue.cc:260] T 209d6a3dc32f43b1b4d536e02bd0ec73 P 631566d8d4dc4a43a1a391ea74418418 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "29f84c922f9f47f8bd7c360af478d359" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 42839 } attrs { replace: true } } peers { permanent_uuid: "631566d8d4dc4a43a1a391ea74418418" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 45397 } attrs { promote: true } }
I20240502 01:55:06.484406 24260 ts_tablet_manager.cc:1398] T 209d6a3dc32f43b1b4d536e02bd0ec73 P 631566d8d4dc4a43a1a391ea74418418: Time spent starting tablet: real 0.002s	user 0.004s	sys 0.000s
I20240502 01:55:06.485297 23168 tablet_copy_service.cc:342] P 29f84c922f9f47f8bd7c360af478d359: Request end of tablet copy session 631566d8d4dc4a43a1a391ea74418418-209d6a3dc32f43b1b4d536e02bd0ec73 received from {username='slave'} at 127.9.176.196:36529
I20240502 01:55:06.485471 23168 tablet_copy_service.cc:433] P 29f84c922f9f47f8bd7c360af478d359: ending tablet copy session 631566d8d4dc4a43a1a391ea74418418-209d6a3dc32f43b1b4d536e02bd0ec73 on tablet 209d6a3dc32f43b1b4d536e02bd0ec73 with peer 631566d8d4dc4a43a1a391ea74418418
I20240502 01:55:06.670269 24008 raft_consensus.cc:1203] T 72b01a904eab46e7865356ab5358acb6 P 631566d8d4dc4a43a1a391ea74418418 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.2->[1.3-1.3]   Dedup: 1.3->[]
I20240502 01:55:06.861356 24008 raft_consensus.cc:1203] T 209d6a3dc32f43b1b4d536e02bd0ec73 P 631566d8d4dc4a43a1a391ea74418418 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20240502 01:55:06.861320 23866 raft_consensus.cc:1203] T 49eddfad6f6e4247a83e82964e8a9470 P 218f63e888ff4c6394af93014b59d65f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.2->[1.3-1.3]   Dedup: 1.3->[]
I20240502 01:55:06.899781 24008 raft_consensus.cc:1203] T 6213c1863b2048c1a281634dd2693c3e P 631566d8d4dc4a43a1a391ea74418418 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20240502 01:55:07.029222 23866 raft_consensus.cc:1203] T 1f4cf78efc5f479c875536136d17fdb0 P 218f63e888ff4c6394af93014b59d65f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20240502 01:55:07.117272 24312 raft_consensus.cc:1050] T 72b01a904eab46e7865356ab5358acb6 P 22b61cbb0f2148db8f62df6a32215c8d: attempting to promote NON_VOTER 631566d8d4dc4a43a1a391ea74418418 to VOTER
I20240502 01:55:07.118144 24312 consensus_queue.cc:237] T 72b01a904eab46e7865356ab5358acb6 P 22b61cbb0f2148db8f62df6a32215c8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "22b61cbb0f2148db8f62df6a32215c8d" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 35853 } attrs { replace: true } } peers { permanent_uuid: "631566d8d4dc4a43a1a391ea74418418" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 45397 } attrs { promote: false } }
I20240502 01:55:07.120270 24008 raft_consensus.cc:1261] T 72b01a904eab46e7865356ab5358acb6 P 631566d8d4dc4a43a1a391ea74418418 [term 1 LEARNER]: Refusing update from remote peer 22b61cbb0f2148db8f62df6a32215c8d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:07.120954 24311 consensus_queue.cc:1035] T 72b01a904eab46e7865356ab5358acb6 P 22b61cbb0f2148db8f62df6a32215c8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "631566d8d4dc4a43a1a391ea74418418" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 45397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:07.124723 24312 raft_consensus.cc:2932] T 72b01a904eab46e7865356ab5358acb6 P 22b61cbb0f2148db8f62df6a32215c8d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, 631566d8d4dc4a43a1a391ea74418418 (127.9.176.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "22b61cbb0f2148db8f62df6a32215c8d" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 35853 } attrs { replace: true } } peers { permanent_uuid: "631566d8d4dc4a43a1a391ea74418418" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 45397 } attrs { promote: false } } }
I20240502 01:55:07.125509 24008 raft_consensus.cc:2932] T 72b01a904eab46e7865356ab5358acb6 P 631566d8d4dc4a43a1a391ea74418418 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, 631566d8d4dc4a43a1a391ea74418418 (127.9.176.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "22b61cbb0f2148db8f62df6a32215c8d" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 35853 } attrs { replace: true } } peers { permanent_uuid: "631566d8d4dc4a43a1a391ea74418418" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 45397 } attrs { promote: false } } }
I20240502 01:55:07.129742 23005 catalog_manager.cc:5484] T 72b01a904eab46e7865356ab5358acb6 P 22b61cbb0f2148db8f62df6a32215c8d reported cstate change: config changed from index 3 to 4, 631566d8d4dc4a43a1a391ea74418418 (127.9.176.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "22b61cbb0f2148db8f62df6a32215c8d" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "22b61cbb0f2148db8f62df6a32215c8d" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 35853 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "631566d8d4dc4a43a1a391ea74418418" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 45397 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:55:07.264773 24340 raft_consensus.cc:1050] T 49eddfad6f6e4247a83e82964e8a9470 P 29f84c922f9f47f8bd7c360af478d359: attempting to promote NON_VOTER 218f63e888ff4c6394af93014b59d65f to VOTER
I20240502 01:55:07.265926 24340 consensus_queue.cc:237] T 49eddfad6f6e4247a83e82964e8a9470 P 29f84c922f9f47f8bd7c360af478d359 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "29f84c922f9f47f8bd7c360af478d359" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 42839 } attrs { replace: true } } peers { permanent_uuid: "218f63e888ff4c6394af93014b59d65f" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 34605 } attrs { promote: false } }
I20240502 01:55:07.269055 23866 raft_consensus.cc:1261] T 49eddfad6f6e4247a83e82964e8a9470 P 218f63e888ff4c6394af93014b59d65f [term 1 LEARNER]: Refusing update from remote peer 29f84c922f9f47f8bd7c360af478d359: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:07.270359 24340 consensus_queue.cc:1035] T 49eddfad6f6e4247a83e82964e8a9470 P 29f84c922f9f47f8bd7c360af478d359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "218f63e888ff4c6394af93014b59d65f" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 34605 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:07.270787 23279 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "72b01a904eab46e7865356ab5358acb6"
dest_uuid: "22b61cbb0f2148db8f62df6a32215c8d"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:60034
I20240502 01:55:07.271082 23279 raft_consensus.cc:595] T 72b01a904eab46e7865356ab5358acb6 P 22b61cbb0f2148db8f62df6a32215c8d [term 1 LEADER]: Received request to transfer leadership
I20240502 01:55:07.277968 24339 raft_consensus.cc:2932] T 49eddfad6f6e4247a83e82964e8a9470 P 29f84c922f9f47f8bd7c360af478d359 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, 218f63e888ff4c6394af93014b59d65f (127.9.176.195) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "29f84c922f9f47f8bd7c360af478d359" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 42839 } attrs { replace: true } } peers { permanent_uuid: "218f63e888ff4c6394af93014b59d65f" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 34605 } attrs { promote: false } } }
I20240502 01:55:07.279228 23866 raft_consensus.cc:2932] T 49eddfad6f6e4247a83e82964e8a9470 P 218f63e888ff4c6394af93014b59d65f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, 218f63e888ff4c6394af93014b59d65f (127.9.176.195) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "29f84c922f9f47f8bd7c360af478d359" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 42839 } attrs { replace: true } } peers { permanent_uuid: "218f63e888ff4c6394af93014b59d65f" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 34605 } attrs { promote: false } } }
I20240502 01:55:07.284929 24311 raft_consensus.cc:1050] T 6213c1863b2048c1a281634dd2693c3e P 22b61cbb0f2148db8f62df6a32215c8d: attempting to promote NON_VOTER 631566d8d4dc4a43a1a391ea74418418 to VOTER
I20240502 01:55:07.284636 23007 catalog_manager.cc:5484] T 49eddfad6f6e4247a83e82964e8a9470 P 29f84c922f9f47f8bd7c360af478d359 reported cstate change: config changed from index 3 to 4, 218f63e888ff4c6394af93014b59d65f (127.9.176.195) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "29f84c922f9f47f8bd7c360af478d359" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "29f84c922f9f47f8bd7c360af478d359" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 42839 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "218f63e888ff4c6394af93014b59d65f" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 34605 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:55:07.285815 24311 consensus_queue.cc:237] T 6213c1863b2048c1a281634dd2693c3e P 22b61cbb0f2148db8f62df6a32215c8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "22b61cbb0f2148db8f62df6a32215c8d" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 35853 } attrs { replace: true } } peers { permanent_uuid: "631566d8d4dc4a43a1a391ea74418418" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 45397 } attrs { promote: false } }
I20240502 01:55:07.288821 24008 raft_consensus.cc:1261] T 6213c1863b2048c1a281634dd2693c3e P 631566d8d4dc4a43a1a391ea74418418 [term 1 LEARNER]: Refusing update from remote peer 22b61cbb0f2148db8f62df6a32215c8d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:55:07.290001 24312 consensus_queue.cc:1035] T 6213c1863b2048c1a281634dd2693c3e P 22b61cbb0f2148db8f62df6a32215c8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "631566d8d4dc4a43a1a391ea74418418" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 45397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:55:07.295302 24311 raft_consensus.cc:2932] T 6213c1863b2048c1a281634dd2693c3e P 22b61cbb0f2148db8f62df6a32215c8d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 631566d8d4dc4a43a1a391ea74418418 (127.9.176.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "22b61cbb0f2148db8f62df6a32215c8d" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 35853 } attrs { replace: true } } peers { permanent_uuid: "631566d8d4dc4a43a1a391ea74418418" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 45397 } attrs { promote: false } } }
I20240502 01:55:07.296300 24008 raft_consensus.cc:2932] T 6213c1863b2048c1a281634dd2693c3e P 631566d8d4dc4a43a1a391ea74418418 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 631566d8d4dc4a43a1a391ea74418418 (127.9.176.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "22b61cbb0f2148db8f62df6a32215c8d" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 35853 } attrs { replace: true } } peers { permanent_uuid: "631566d8d4dc4a43a1a391ea74418418" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 45397 } attrs { promote: false } } }
I20240502 01:55:07.300573 23005 catalog_manager.cc:5484] T 6213c1863b2048c1a281634dd2693c3e P 22b61cbb0f2148db8f62df6a32215c8d reported cstate change: config changed from index 2 to 3, 631566d8d4dc4a43a1a391ea74418418 (127.9.176.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "22b61cbb0f2148db8f62df6a32215c8d" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "22b61cbb0f2148db8f62df6a32215c8d" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 35853 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "631566d8d4dc4a43a1a391ea74418418" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 45397 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:55:07.323529 23279 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6213c1863b2048c1a281634dd2693c3e"
dest_uuid: "22b61cbb0f2148db8f62df6a32215c8d"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:60056
I20240502 01:55:07.323808 23279 raft_consensus.cc:595] T 6213c1863b2048c1a281634dd2693c3e P 22b61cbb0f2148db8f62df6a32215c8d [term 1 LEADER]: Received request to transfer leadership
I20240502 01:55:07.345033 23148 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "49eddfad6f6e4247a83e82964e8a9470"
dest_uuid: "29f84c922f9f47f8bd7c360af478d359"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:52756
I20240502 01:55:07.345324 23148 raft_consensus.cc:595] T 49eddfad6f6e4247a83e82964e8a9470 P 29f84c922f9f47f8bd7c360af478d359 [term 1 LEADER]: Received request to transfer leadership
I20240502 01:55:07.364332 24340 raft_consensus.cc:1050] T 209d6a3dc32f43b1b4d536e02bd0ec73 P 29f84c922f9f47f8bd7c360af478d359: attempting to promote NON_VOTER 631566d8d4dc4a43a1a391ea74418418 to VOTER
I20240502 01:55:07.365054 24340 consensus_queue.cc:237] T 209d6a3dc32f43b1b4d536e02bd0ec73 P 29f84c922f9f47f8bd7c360af478d359 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "29f84c922f9f47f8bd7c360af478d359" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 42839 } attrs { replace: true } } peers { permanent_uuid: "631566d8d4dc4a43a1a391ea74418418" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 45397 } attrs { promote: false } }
I20240502 01:55:07.367111 24008 raft_consensus.cc:1261] T 209d6a3dc32f43b1b4d536e02bd0ec73 P 631566d8d4dc4a43a1a391ea74418418 [term 1 LEARNER]: Refusing update from remote peer 29f84c922f9f47f8bd7c360af478d359: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:55:07.367800 24339 consensus_queue.cc:1035] T 209d6a3dc32f43b1b4d536e02bd0ec73 P 29f84c922f9f47f8bd7c360af478d359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "631566d8d4dc4a43a1a391ea74418418" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 45397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:55:07.370879 24340 raft_consensus.cc:2932] T 209d6a3dc32f43b1b4d536e02bd0ec73 P 29f84c922f9f47f8bd7c360af478d359 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 631566d8d4dc4a43a1a391ea74418418 (127.9.176.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "29f84c922f9f47f8bd7c360af478d359" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 42839 } attrs { replace: true } } peers { permanent_uuid: "631566d8d4dc4a43a1a391ea74418418" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 45397 } attrs { promote: false } } }
I20240502 01:55:07.371629 24008 raft_consensus.cc:2932] T 209d6a3dc32f43b1b4d536e02bd0ec73 P 631566d8d4dc4a43a1a391ea74418418 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 631566d8d4dc4a43a1a391ea74418418 (127.9.176.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "29f84c922f9f47f8bd7c360af478d359" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 42839 } attrs { replace: true } } peers { permanent_uuid: "631566d8d4dc4a43a1a391ea74418418" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 45397 } attrs { promote: false } } }
I20240502 01:55:07.375792 23005 catalog_manager.cc:5484] T 209d6a3dc32f43b1b4d536e02bd0ec73 P 29f84c922f9f47f8bd7c360af478d359 reported cstate change: config changed from index 2 to 3, 631566d8d4dc4a43a1a391ea74418418 (127.9.176.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "29f84c922f9f47f8bd7c360af478d359" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "29f84c922f9f47f8bd7c360af478d359" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 42839 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "631566d8d4dc4a43a1a391ea74418418" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 45397 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:55:07.421475 24338 raft_consensus.cc:1050] T 1f4cf78efc5f479c875536136d17fdb0 P 29f84c922f9f47f8bd7c360af478d359: attempting to promote NON_VOTER 218f63e888ff4c6394af93014b59d65f to VOTER
I20240502 01:55:07.422111 24338 consensus_queue.cc:237] T 1f4cf78efc5f479c875536136d17fdb0 P 29f84c922f9f47f8bd7c360af478d359 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "29f84c922f9f47f8bd7c360af478d359" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 42839 } attrs { replace: true } } peers { permanent_uuid: "218f63e888ff4c6394af93014b59d65f" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 34605 } attrs { promote: false } }
I20240502 01:55:07.424311 23866 raft_consensus.cc:1261] T 1f4cf78efc5f479c875536136d17fdb0 P 218f63e888ff4c6394af93014b59d65f [term 1 LEARNER]: Refusing update from remote peer 29f84c922f9f47f8bd7c360af478d359: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:55:07.424898 24339 consensus_queue.cc:1035] T 1f4cf78efc5f479c875536136d17fdb0 P 29f84c922f9f47f8bd7c360af478d359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "218f63e888ff4c6394af93014b59d65f" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 34605 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:55:07.427693 24338 raft_consensus.cc:2932] T 1f4cf78efc5f479c875536136d17fdb0 P 29f84c922f9f47f8bd7c360af478d359 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 218f63e888ff4c6394af93014b59d65f (127.9.176.195) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "29f84c922f9f47f8bd7c360af478d359" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 42839 } attrs { replace: true } } peers { permanent_uuid: "218f63e888ff4c6394af93014b59d65f" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 34605 } attrs { promote: false } } }
I20240502 01:55:07.428468 23866 raft_consensus.cc:2932] T 1f4cf78efc5f479c875536136d17fdb0 P 218f63e888ff4c6394af93014b59d65f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 218f63e888ff4c6394af93014b59d65f (127.9.176.195) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "29f84c922f9f47f8bd7c360af478d359" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 42839 } attrs { replace: true } } peers { permanent_uuid: "218f63e888ff4c6394af93014b59d65f" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 34605 } attrs { promote: false } } }
I20240502 01:55:07.433437 23007 catalog_manager.cc:5484] T 1f4cf78efc5f479c875536136d17fdb0 P 29f84c922f9f47f8bd7c360af478d359 reported cstate change: config changed from index 2 to 3, 218f63e888ff4c6394af93014b59d65f (127.9.176.195) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "29f84c922f9f47f8bd7c360af478d359" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "29f84c922f9f47f8bd7c360af478d359" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 42839 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "218f63e888ff4c6394af93014b59d65f" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 34605 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:55:07.566388 23148 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1f4cf78efc5f479c875536136d17fdb0"
dest_uuid: "29f84c922f9f47f8bd7c360af478d359"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:52778
I20240502 01:55:07.566697 23148 raft_consensus.cc:595] T 1f4cf78efc5f479c875536136d17fdb0 P 29f84c922f9f47f8bd7c360af478d359 [term 1 LEADER]: Received request to transfer leadership
I20240502 01:55:07.587181 23279 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "72b01a904eab46e7865356ab5358acb6"
dest_uuid: "22b61cbb0f2148db8f62df6a32215c8d"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:60076
I20240502 01:55:07.587428 23279 raft_consensus.cc:595] T 72b01a904eab46e7865356ab5358acb6 P 22b61cbb0f2148db8f62df6a32215c8d [term 1 LEADER]: Received request to transfer leadership
I20240502 01:55:07.608165 23148 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "209d6a3dc32f43b1b4d536e02bd0ec73"
dest_uuid: "29f84c922f9f47f8bd7c360af478d359"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:52804
I20240502 01:55:07.608407 23148 raft_consensus.cc:595] T 209d6a3dc32f43b1b4d536e02bd0ec73 P 29f84c922f9f47f8bd7c360af478d359 [term 1 LEADER]: Received request to transfer leadership
I20240502 01:55:07.628630 23279 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6213c1863b2048c1a281634dd2693c3e"
dest_uuid: "22b61cbb0f2148db8f62df6a32215c8d"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:60098
I20240502 01:55:07.628877 23279 raft_consensus.cc:595] T 6213c1863b2048c1a281634dd2693c3e P 22b61cbb0f2148db8f62df6a32215c8d [term 1 LEADER]: Received request to transfer leadership
I20240502 01:55:07.648149 23148 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "49eddfad6f6e4247a83e82964e8a9470"
dest_uuid: "29f84c922f9f47f8bd7c360af478d359"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:52824
I20240502 01:55:07.648389 23148 raft_consensus.cc:595] T 49eddfad6f6e4247a83e82964e8a9470 P 29f84c922f9f47f8bd7c360af478d359 [term 1 LEADER]: Received request to transfer leadership
I20240502 01:55:07.661087 24311 raft_consensus.cc:979] T 72b01a904eab46e7865356ab5358acb6 P 22b61cbb0f2148db8f62df6a32215c8d: : Instructing follower 631566d8d4dc4a43a1a391ea74418418 to start an election
I20240502 01:55:07.661279 24311 raft_consensus.cc:1067] T 72b01a904eab46e7865356ab5358acb6 P 22b61cbb0f2148db8f62df6a32215c8d [term 1 LEADER]: Signalling peer 631566d8d4dc4a43a1a391ea74418418 to start an election
I20240502 01:55:07.662326 24008 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "72b01a904eab46e7865356ab5358acb6"
dest_uuid: "631566d8d4dc4a43a1a391ea74418418"
 from {username='slave'} at 127.9.176.194:40281
I20240502 01:55:07.662680 24008 raft_consensus.cc:482] T 72b01a904eab46e7865356ab5358acb6 P 631566d8d4dc4a43a1a391ea74418418 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:55:07.662848 24008 raft_consensus.cc:3037] T 72b01a904eab46e7865356ab5358acb6 P 631566d8d4dc4a43a1a391ea74418418 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:55:07.665002 24008 raft_consensus.cc:504] T 72b01a904eab46e7865356ab5358acb6 P 631566d8d4dc4a43a1a391ea74418418 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "22b61cbb0f2148db8f62df6a32215c8d" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 35853 } attrs { replace: true } } peers { permanent_uuid: "631566d8d4dc4a43a1a391ea74418418" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 45397 } attrs { promote: false } }
I20240502 01:55:07.666339 24008 leader_election.cc:288] T 72b01a904eab46e7865356ab5358acb6 P 631566d8d4dc4a43a1a391ea74418418 [CANDIDATE]: Term 2 election: Requested vote from peers 22b61cbb0f2148db8f62df6a32215c8d (127.9.176.194:35853)
I20240502 01:55:07.667098 23279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72b01a904eab46e7865356ab5358acb6" candidate_uuid: "631566d8d4dc4a43a1a391ea74418418" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "22b61cbb0f2148db8f62df6a32215c8d"
I20240502 01:55:07.667407 23279 raft_consensus.cc:3032] T 72b01a904eab46e7865356ab5358acb6 P 22b61cbb0f2148db8f62df6a32215c8d [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:55:07.667531 23279 raft_consensus.cc:729] T 72b01a904eab46e7865356ab5358acb6 P 22b61cbb0f2148db8f62df6a32215c8d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 22b61cbb0f2148db8f62df6a32215c8d, State: Running, Role: LEADER
I20240502 01:55:07.667886 23279 consensus_queue.cc:260] T 72b01a904eab46e7865356ab5358acb6 P 22b61cbb0f2148db8f62df6a32215c8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "22b61cbb0f2148db8f62df6a32215c8d" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 35853 } attrs { replace: true } } peers { permanent_uuid: "631566d8d4dc4a43a1a391ea74418418" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 45397 } attrs { promote: false } }
I20240502 01:55:07.668258 23279 raft_consensus.cc:3037] T 72b01a904eab46e7865356ab5358acb6 P 22b61cbb0f2148db8f62df6a32215c8d [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:55:07.670341 23279 raft_consensus.cc:2454] T 72b01a904eab46e7865356ab5358acb6 P 22b61cbb0f2148db8f62df6a32215c8d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 631566d8d4dc4a43a1a391ea74418418 in term 2.
I20240502 01:55:07.671080 23942 leader_election.cc:302] T 72b01a904eab46e7865356ab5358acb6 P 631566d8d4dc4a43a1a391ea74418418 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 22b61cbb0f2148db8f62df6a32215c8d, 631566d8d4dc4a43a1a391ea74418418; no voters: 
I20240502 01:55:07.671519 24269 raft_consensus.cc:2781] T 72b01a904eab46e7865356ab5358acb6 P 631566d8d4dc4a43a1a391ea74418418 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:55:07.672130 24269 raft_consensus.cc:686] T 72b01a904eab46e7865356ab5358acb6 P 631566d8d4dc4a43a1a391ea74418418 [term 2 LEADER]: Becoming Leader. State: Replica: 631566d8d4dc4a43a1a391ea74418418, State: Running, Role: LEADER
I20240502 01:55:07.672600 24269 consensus_queue.cc:237] T 72b01a904eab46e7865356ab5358acb6 P 631566d8d4dc4a43a1a391ea74418418 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "22b61cbb0f2148db8f62df6a32215c8d" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 35853 } attrs { replace: true } } peers { permanent_uuid: "631566d8d4dc4a43a1a391ea74418418" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 45397 } attrs { promote: false } }
I20240502 01:55:07.676362 23005 catalog_manager.cc:5484] T 72b01a904eab46e7865356ab5358acb6 P 631566d8d4dc4a43a1a391ea74418418 reported cstate change: term changed from 1 to 2, leader changed from 22b61cbb0f2148db8f62df6a32215c8d (127.9.176.194) to 631566d8d4dc4a43a1a391ea74418418 (127.9.176.196). New cstate: current_term: 2 leader_uuid: "631566d8d4dc4a43a1a391ea74418418" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "22b61cbb0f2148db8f62df6a32215c8d" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 35853 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "631566d8d4dc4a43a1a391ea74418418" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 45397 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240502 01:55:07.688620 22992 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet 72b01a904eab46e7865356ab5358acb6 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER failed with leader 631566d8d4dc4a43a1a391ea74418418 (127.9.176.196:45397) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term
I20240502 01:55:07.761243 24338 raft_consensus.cc:979] T 49eddfad6f6e4247a83e82964e8a9470 P 29f84c922f9f47f8bd7c360af478d359: : Instructing follower 218f63e888ff4c6394af93014b59d65f to start an election
I20240502 01:55:07.761466 24338 raft_consensus.cc:1067] T 49eddfad6f6e4247a83e82964e8a9470 P 29f84c922f9f47f8bd7c360af478d359 [term 1 LEADER]: Signalling peer 218f63e888ff4c6394af93014b59d65f to start an election
I20240502 01:55:07.762404 23866 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "49eddfad6f6e4247a83e82964e8a9470"
dest_uuid: "218f63e888ff4c6394af93014b59d65f"
 from {username='slave'} at 127.9.176.193:57531
I20240502 01:55:07.762696 23866 raft_consensus.cc:482] T 49eddfad6f6e4247a83e82964e8a9470 P 218f63e888ff4c6394af93014b59d65f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:55:07.762840 23866 raft_consensus.cc:3037] T 49eddfad6f6e4247a83e82964e8a9470 P 218f63e888ff4c6394af93014b59d65f [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:55:07.766052 23866 raft_consensus.cc:504] T 49eddfad6f6e4247a83e82964e8a9470 P 218f63e888ff4c6394af93014b59d65f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "29f84c922f9f47f8bd7c360af478d359" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 42839 } attrs { replace: true } } peers { permanent_uuid: "218f63e888ff4c6394af93014b59d65f" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 34605 } attrs { promote: false } }
I20240502 01:55:07.767231 23866 leader_election.cc:288] T 49eddfad6f6e4247a83e82964e8a9470 P 218f63e888ff4c6394af93014b59d65f [CANDIDATE]: Term 2 election: Requested vote from peers 29f84c922f9f47f8bd7c360af478d359 (127.9.176.193:42839)
I20240502 01:55:07.768113 23148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49eddfad6f6e4247a83e82964e8a9470" candidate_uuid: "218f63e888ff4c6394af93014b59d65f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "29f84c922f9f47f8bd7c360af478d359"
I20240502 01:55:07.768443 23148 raft_consensus.cc:3032] T 49eddfad6f6e4247a83e82964e8a9470 P 29f84c922f9f47f8bd7c360af478d359 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:55:07.768537 23148 raft_consensus.cc:729] T 49eddfad6f6e4247a83e82964e8a9470 P 29f84c922f9f47f8bd7c360af478d359 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 29f84c922f9f47f8bd7c360af478d359, State: Running, Role: LEADER
I20240502 01:55:07.768890 23148 consensus_queue.cc:260] T 49eddfad6f6e4247a83e82964e8a9470 P 29f84c922f9f47f8bd7c360af478d359 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "29f84c922f9f47f8bd7c360af478d359" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 42839 } attrs { replace: true } } peers { permanent_uuid: "218f63e888ff4c6394af93014b59d65f" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 34605 } attrs { promote: false } }
I20240502 01:55:07.769250 23148 raft_consensus.cc:3037] T 49eddfad6f6e4247a83e82964e8a9470 P 29f84c922f9f47f8bd7c360af478d359 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:55:07.772222 23148 raft_consensus.cc:2454] T 49eddfad6f6e4247a83e82964e8a9470 P 29f84c922f9f47f8bd7c360af478d359 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 218f63e888ff4c6394af93014b59d65f in term 2.
I20240502 01:55:07.773005 23799 leader_election.cc:302] T 49eddfad6f6e4247a83e82964e8a9470 P 218f63e888ff4c6394af93014b59d65f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 218f63e888ff4c6394af93014b59d65f, 29f84c922f9f47f8bd7c360af478d359; no voters: 
I20240502 01:55:07.773464 24267 raft_consensus.cc:2781] T 49eddfad6f6e4247a83e82964e8a9470 P 218f63e888ff4c6394af93014b59d65f [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:55:07.773929 24267 raft_consensus.cc:686] T 49eddfad6f6e4247a83e82964e8a9470 P 218f63e888ff4c6394af93014b59d65f [term 2 LEADER]: Becoming Leader. State: Replica: 218f63e888ff4c6394af93014b59d65f, State: Running, Role: LEADER
I20240502 01:55:07.774296 24267 consensus_queue.cc:237] T 49eddfad6f6e4247a83e82964e8a9470 P 218f63e888ff4c6394af93014b59d65f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "29f84c922f9f47f8bd7c360af478d359" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 42839 } attrs { replace: true } } peers { permanent_uuid: "218f63e888ff4c6394af93014b59d65f" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 34605 } attrs { promote: false } }
I20240502 01:55:07.775437 24312 raft_consensus.cc:979] T 6213c1863b2048c1a281634dd2693c3e P 22b61cbb0f2148db8f62df6a32215c8d: : Instructing follower 631566d8d4dc4a43a1a391ea74418418 to start an election
I20240502 01:55:07.775645 24312 raft_consensus.cc:1067] T 6213c1863b2048c1a281634dd2693c3e P 22b61cbb0f2148db8f62df6a32215c8d [term 1 LEADER]: Signalling peer 631566d8d4dc4a43a1a391ea74418418 to start an election
I20240502 01:55:07.776600 24008 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6213c1863b2048c1a281634dd2693c3e"
dest_uuid: "631566d8d4dc4a43a1a391ea74418418"
 from {username='slave'} at 127.9.176.194:40281
I20240502 01:55:07.776829 24008 raft_consensus.cc:482] T 6213c1863b2048c1a281634dd2693c3e P 631566d8d4dc4a43a1a391ea74418418 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:55:07.776988 24008 raft_consensus.cc:3037] T 6213c1863b2048c1a281634dd2693c3e P 631566d8d4dc4a43a1a391ea74418418 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:55:07.778102 23005 catalog_manager.cc:5484] T 49eddfad6f6e4247a83e82964e8a9470 P 218f63e888ff4c6394af93014b59d65f reported cstate change: term changed from 1 to 2, leader changed from 29f84c922f9f47f8bd7c360af478d359 (127.9.176.193) to 218f63e888ff4c6394af93014b59d65f (127.9.176.195). New cstate: current_term: 2 leader_uuid: "218f63e888ff4c6394af93014b59d65f" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "29f84c922f9f47f8bd7c360af478d359" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 42839 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "218f63e888ff4c6394af93014b59d65f" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 34605 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:55:07.780222 24008 raft_consensus.cc:504] T 6213c1863b2048c1a281634dd2693c3e P 631566d8d4dc4a43a1a391ea74418418 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "22b61cbb0f2148db8f62df6a32215c8d" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 35853 } attrs { replace: true } } peers { permanent_uuid: "631566d8d4dc4a43a1a391ea74418418" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 45397 } attrs { promote: false } }
I20240502 01:55:07.780870 24008 leader_election.cc:288] T 6213c1863b2048c1a281634dd2693c3e P 631566d8d4dc4a43a1a391ea74418418 [CANDIDATE]: Term 2 election: Requested vote from peers 22b61cbb0f2148db8f62df6a32215c8d (127.9.176.194:35853)
I20240502 01:55:07.781677 23279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6213c1863b2048c1a281634dd2693c3e" candidate_uuid: "631566d8d4dc4a43a1a391ea74418418" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "22b61cbb0f2148db8f62df6a32215c8d"
I20240502 01:55:07.781899 23279 raft_consensus.cc:3032] T 6213c1863b2048c1a281634dd2693c3e P 22b61cbb0f2148db8f62df6a32215c8d [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:55:07.782027 23279 raft_consensus.cc:729] T 6213c1863b2048c1a281634dd2693c3e P 22b61cbb0f2148db8f62df6a32215c8d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 22b61cbb0f2148db8f62df6a32215c8d, State: Running, Role: LEADER
I20240502 01:55:07.782284 23279 consensus_queue.cc:260] T 6213c1863b2048c1a281634dd2693c3e P 22b61cbb0f2148db8f62df6a32215c8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "22b61cbb0f2148db8f62df6a32215c8d" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 35853 } attrs { replace: true } } peers { permanent_uuid: "631566d8d4dc4a43a1a391ea74418418" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 45397 } attrs { promote: false } }
I20240502 01:55:07.782586 23279 raft_consensus.cc:3037] T 6213c1863b2048c1a281634dd2693c3e P 22b61cbb0f2148db8f62df6a32215c8d [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:55:07.785574 23279 raft_consensus.cc:2454] T 6213c1863b2048c1a281634dd2693c3e P 22b61cbb0f2148db8f62df6a32215c8d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 631566d8d4dc4a43a1a391ea74418418 in term 2.
I20240502 01:55:07.786128 23942 leader_election.cc:302] T 6213c1863b2048c1a281634dd2693c3e P 631566d8d4dc4a43a1a391ea74418418 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 22b61cbb0f2148db8f62df6a32215c8d, 631566d8d4dc4a43a1a391ea74418418; no voters: 
I20240502 01:55:07.786473 24269 raft_consensus.cc:2781] T 6213c1863b2048c1a281634dd2693c3e P 631566d8d4dc4a43a1a391ea74418418 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:55:07.786653 24269 raft_consensus.cc:686] T 6213c1863b2048c1a281634dd2693c3e P 631566d8d4dc4a43a1a391ea74418418 [term 2 LEADER]: Becoming Leader. State: Replica: 631566d8d4dc4a43a1a391ea74418418, State: Running, Role: LEADER
I20240502 01:55:07.786975 24269 consensus_queue.cc:237] T 6213c1863b2048c1a281634dd2693c3e P 631566d8d4dc4a43a1a391ea74418418 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "22b61cbb0f2148db8f62df6a32215c8d" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 35853 } attrs { replace: true } } peers { permanent_uuid: "631566d8d4dc4a43a1a391ea74418418" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 45397 } attrs { promote: false } }
I20240502 01:55:07.789613 23005 catalog_manager.cc:5484] T 6213c1863b2048c1a281634dd2693c3e P 631566d8d4dc4a43a1a391ea74418418 reported cstate change: term changed from 1 to 2, leader changed from 22b61cbb0f2148db8f62df6a32215c8d (127.9.176.194) to 631566d8d4dc4a43a1a391ea74418418 (127.9.176.196). New cstate: current_term: 2 leader_uuid: "631566d8d4dc4a43a1a391ea74418418" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "22b61cbb0f2148db8f62df6a32215c8d" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 35853 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "631566d8d4dc4a43a1a391ea74418418" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 45397 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:55:07.802542 24340 raft_consensus.cc:979] T 209d6a3dc32f43b1b4d536e02bd0ec73 P 29f84c922f9f47f8bd7c360af478d359: : Instructing follower 631566d8d4dc4a43a1a391ea74418418 to start an election
I20240502 01:55:07.802753 24340 raft_consensus.cc:1067] T 209d6a3dc32f43b1b4d536e02bd0ec73 P 29f84c922f9f47f8bd7c360af478d359 [term 1 LEADER]: Signalling peer 631566d8d4dc4a43a1a391ea74418418 to start an election
I20240502 01:55:07.803494 24007 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "209d6a3dc32f43b1b4d536e02bd0ec73"
dest_uuid: "631566d8d4dc4a43a1a391ea74418418"
 from {username='slave'} at 127.9.176.193:32807
I20240502 01:55:07.803820 24007 raft_consensus.cc:482] T 209d6a3dc32f43b1b4d536e02bd0ec73 P 631566d8d4dc4a43a1a391ea74418418 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:55:07.803968 24007 raft_consensus.cc:3037] T 209d6a3dc32f43b1b4d536e02bd0ec73 P 631566d8d4dc4a43a1a391ea74418418 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:55:07.806066 24007 raft_consensus.cc:504] T 209d6a3dc32f43b1b4d536e02bd0ec73 P 631566d8d4dc4a43a1a391ea74418418 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "29f84c922f9f47f8bd7c360af478d359" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 42839 } attrs { replace: true } } peers { permanent_uuid: "631566d8d4dc4a43a1a391ea74418418" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 45397 } attrs { promote: false } }
I20240502 01:55:07.806771 24007 leader_election.cc:288] T 209d6a3dc32f43b1b4d536e02bd0ec73 P 631566d8d4dc4a43a1a391ea74418418 [CANDIDATE]: Term 2 election: Requested vote from peers 29f84c922f9f47f8bd7c360af478d359 (127.9.176.193:42839)
I20240502 01:55:07.807399 23148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "209d6a3dc32f43b1b4d536e02bd0ec73" candidate_uuid: "631566d8d4dc4a43a1a391ea74418418" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "29f84c922f9f47f8bd7c360af478d359"
I20240502 01:55:07.807649 23148 raft_consensus.cc:3032] T 209d6a3dc32f43b1b4d536e02bd0ec73 P 29f84c922f9f47f8bd7c360af478d359 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:55:07.807746 23148 raft_consensus.cc:729] T 209d6a3dc32f43b1b4d536e02bd0ec73 P 29f84c922f9f47f8bd7c360af478d359 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 29f84c922f9f47f8bd7c360af478d359, State: Running, Role: LEADER
I20240502 01:55:07.808063 23148 consensus_queue.cc:260] T 209d6a3dc32f43b1b4d536e02bd0ec73 P 29f84c922f9f47f8bd7c360af478d359 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "29f84c922f9f47f8bd7c360af478d359" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 42839 } attrs { replace: true } } peers { permanent_uuid: "631566d8d4dc4a43a1a391ea74418418" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 45397 } attrs { promote: false } }
I20240502 01:55:07.808341 23148 raft_consensus.cc:3037] T 209d6a3dc32f43b1b4d536e02bd0ec73 P 29f84c922f9f47f8bd7c360af478d359 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:55:07.810396 23148 raft_consensus.cc:2454] T 209d6a3dc32f43b1b4d536e02bd0ec73 P 29f84c922f9f47f8bd7c360af478d359 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 631566d8d4dc4a43a1a391ea74418418 in term 2.
I20240502 01:55:07.810935 23942 leader_election.cc:302] T 209d6a3dc32f43b1b4d536e02bd0ec73 P 631566d8d4dc4a43a1a391ea74418418 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 29f84c922f9f47f8bd7c360af478d359, 631566d8d4dc4a43a1a391ea74418418; no voters: 
I20240502 01:55:07.811290 24472 raft_consensus.cc:2781] T 209d6a3dc32f43b1b4d536e02bd0ec73 P 631566d8d4dc4a43a1a391ea74418418 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:55:07.811442 24472 raft_consensus.cc:686] T 209d6a3dc32f43b1b4d536e02bd0ec73 P 631566d8d4dc4a43a1a391ea74418418 [term 2 LEADER]: Becoming Leader. State: Replica: 631566d8d4dc4a43a1a391ea74418418, State: Running, Role: LEADER
I20240502 01:55:07.811816 24472 consensus_queue.cc:237] T 209d6a3dc32f43b1b4d536e02bd0ec73 P 631566d8d4dc4a43a1a391ea74418418 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "29f84c922f9f47f8bd7c360af478d359" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 42839 } attrs { replace: true } } peers { permanent_uuid: "631566d8d4dc4a43a1a391ea74418418" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 45397 } attrs { promote: false } }
I20240502 01:55:07.814340 23005 catalog_manager.cc:5484] T 209d6a3dc32f43b1b4d536e02bd0ec73 P 631566d8d4dc4a43a1a391ea74418418 reported cstate change: term changed from 1 to 2, leader changed from 29f84c922f9f47f8bd7c360af478d359 (127.9.176.193) to 631566d8d4dc4a43a1a391ea74418418 (127.9.176.196). New cstate: current_term: 2 leader_uuid: "631566d8d4dc4a43a1a391ea74418418" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "29f84c922f9f47f8bd7c360af478d359" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 42839 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "631566d8d4dc4a43a1a391ea74418418" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 45397 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:55:07.875182 23148 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1f4cf78efc5f479c875536136d17fdb0"
dest_uuid: "29f84c922f9f47f8bd7c360af478d359"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:52854
I20240502 01:55:07.875430 23148 raft_consensus.cc:595] T 1f4cf78efc5f479c875536136d17fdb0 P 29f84c922f9f47f8bd7c360af478d359 [term 1 LEADER]: Received request to transfer leadership
I20240502 01:55:08.014521 24338 raft_consensus.cc:979] T 1f4cf78efc5f479c875536136d17fdb0 P 29f84c922f9f47f8bd7c360af478d359: : Instructing follower 218f63e888ff4c6394af93014b59d65f to start an election
I20240502 01:55:08.014714 24338 raft_consensus.cc:1067] T 1f4cf78efc5f479c875536136d17fdb0 P 29f84c922f9f47f8bd7c360af478d359 [term 1 LEADER]: Signalling peer 218f63e888ff4c6394af93014b59d65f to start an election
I20240502 01:55:08.015306 23866 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1f4cf78efc5f479c875536136d17fdb0"
dest_uuid: "218f63e888ff4c6394af93014b59d65f"
 from {username='slave'} at 127.9.176.193:57531
I20240502 01:55:08.015520 23866 raft_consensus.cc:482] T 1f4cf78efc5f479c875536136d17fdb0 P 218f63e888ff4c6394af93014b59d65f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:55:08.015645 23866 raft_consensus.cc:3037] T 1f4cf78efc5f479c875536136d17fdb0 P 218f63e888ff4c6394af93014b59d65f [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:55:08.017638 23866 raft_consensus.cc:504] T 1f4cf78efc5f479c875536136d17fdb0 P 218f63e888ff4c6394af93014b59d65f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "29f84c922f9f47f8bd7c360af478d359" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 42839 } attrs { replace: true } } peers { permanent_uuid: "218f63e888ff4c6394af93014b59d65f" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 34605 } attrs { promote: false } }
I20240502 01:55:08.018159 23866 leader_election.cc:288] T 1f4cf78efc5f479c875536136d17fdb0 P 218f63e888ff4c6394af93014b59d65f [CANDIDATE]: Term 2 election: Requested vote from peers 29f84c922f9f47f8bd7c360af478d359 (127.9.176.193:42839)
I20240502 01:55:08.018826 23148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f4cf78efc5f479c875536136d17fdb0" candidate_uuid: "218f63e888ff4c6394af93014b59d65f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "29f84c922f9f47f8bd7c360af478d359"
I20240502 01:55:08.019061 23148 raft_consensus.cc:3032] T 1f4cf78efc5f479c875536136d17fdb0 P 29f84c922f9f47f8bd7c360af478d359 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:55:08.019129 23148 raft_consensus.cc:729] T 1f4cf78efc5f479c875536136d17fdb0 P 29f84c922f9f47f8bd7c360af478d359 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 29f84c922f9f47f8bd7c360af478d359, State: Running, Role: LEADER
I20240502 01:55:08.019367 23148 consensus_queue.cc:260] T 1f4cf78efc5f479c875536136d17fdb0 P 29f84c922f9f47f8bd7c360af478d359 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "29f84c922f9f47f8bd7c360af478d359" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 42839 } attrs { replace: true } } peers { permanent_uuid: "218f63e888ff4c6394af93014b59d65f" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 34605 } attrs { promote: false } }
I20240502 01:55:08.019654 23148 raft_consensus.cc:3037] T 1f4cf78efc5f479c875536136d17fdb0 P 29f84c922f9f47f8bd7c360af478d359 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:55:08.021580 23148 raft_consensus.cc:2454] T 1f4cf78efc5f479c875536136d17fdb0 P 29f84c922f9f47f8bd7c360af478d359 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 218f63e888ff4c6394af93014b59d65f in term 2.
I20240502 01:55:08.022118 23799 leader_election.cc:302] T 1f4cf78efc5f479c875536136d17fdb0 P 218f63e888ff4c6394af93014b59d65f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 218f63e888ff4c6394af93014b59d65f, 29f84c922f9f47f8bd7c360af478d359; no voters: 
I20240502 01:55:08.022411 24267 raft_consensus.cc:2781] T 1f4cf78efc5f479c875536136d17fdb0 P 218f63e888ff4c6394af93014b59d65f [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:55:08.022572 24267 raft_consensus.cc:686] T 1f4cf78efc5f479c875536136d17fdb0 P 218f63e888ff4c6394af93014b59d65f [term 2 LEADER]: Becoming Leader. State: Replica: 218f63e888ff4c6394af93014b59d65f, State: Running, Role: LEADER
I20240502 01:55:08.022817 24267 consensus_queue.cc:237] T 1f4cf78efc5f479c875536136d17fdb0 P 218f63e888ff4c6394af93014b59d65f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "29f84c922f9f47f8bd7c360af478d359" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 42839 } attrs { replace: true } } peers { permanent_uuid: "218f63e888ff4c6394af93014b59d65f" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 34605 } attrs { promote: false } }
I20240502 01:55:08.025240 23005 catalog_manager.cc:5484] T 1f4cf78efc5f479c875536136d17fdb0 P 218f63e888ff4c6394af93014b59d65f reported cstate change: term changed from 1 to 2, leader changed from 29f84c922f9f47f8bd7c360af478d359 (127.9.176.193) to 218f63e888ff4c6394af93014b59d65f (127.9.176.195). New cstate: current_term: 2 leader_uuid: "218f63e888ff4c6394af93014b59d65f" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "29f84c922f9f47f8bd7c360af478d359" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 42839 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "218f63e888ff4c6394af93014b59d65f" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 34605 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:55:08.191500 23279 raft_consensus.cc:1261] T 72b01a904eab46e7865356ab5358acb6 P 22b61cbb0f2148db8f62df6a32215c8d [term 2 FOLLOWER]: Refusing update from remote peer 631566d8d4dc4a43a1a391ea74418418: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 01:55:08.192418 24472 consensus_queue.cc:1035] T 72b01a904eab46e7865356ab5358acb6 P 631566d8d4dc4a43a1a391ea74418418 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22b61cbb0f2148db8f62df6a32215c8d" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 35853 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:08.233117 23148 raft_consensus.cc:1261] T 209d6a3dc32f43b1b4d536e02bd0ec73 P 29f84c922f9f47f8bd7c360af478d359 [term 2 FOLLOWER]: Refusing update from remote peer 631566d8d4dc4a43a1a391ea74418418: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 01:55:08.233912 24472 consensus_queue.cc:1035] T 209d6a3dc32f43b1b4d536e02bd0ec73 P 631566d8d4dc4a43a1a391ea74418418 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29f84c922f9f47f8bd7c360af478d359" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 42839 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:08.250304 23279 raft_consensus.cc:1261] T 6213c1863b2048c1a281634dd2693c3e P 22b61cbb0f2148db8f62df6a32215c8d [term 2 FOLLOWER]: Refusing update from remote peer 631566d8d4dc4a43a1a391ea74418418: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 01:55:08.250969 24269 consensus_queue.cc:1035] T 6213c1863b2048c1a281634dd2693c3e P 631566d8d4dc4a43a1a391ea74418418 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22b61cbb0f2148db8f62df6a32215c8d" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 35853 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:08.343243 24007 consensus_queue.cc:237] T 72b01a904eab46e7865356ab5358acb6 P 631566d8d4dc4a43a1a391ea74418418 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "631566d8d4dc4a43a1a391ea74418418" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 45397 } attrs { promote: false } }
I20240502 01:55:08.344321 24269 raft_consensus.cc:2932] T 72b01a904eab46e7865356ab5358acb6 P 631566d8d4dc4a43a1a391ea74418418 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 22b61cbb0f2148db8f62df6a32215c8d (127.9.176.194) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "631566d8d4dc4a43a1a391ea74418418" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 45397 } attrs { promote: false } } }
I20240502 01:55:08.347227 22992 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 72b01a904eab46e7865356ab5358acb6 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240502 01:55:08.348618 23007 catalog_manager.cc:5484] T 72b01a904eab46e7865356ab5358acb6 P 631566d8d4dc4a43a1a391ea74418418 reported cstate change: config changed from index 4 to 6, VOTER 22b61cbb0f2148db8f62df6a32215c8d (127.9.176.194) evicted. New cstate: current_term: 2 leader_uuid: "631566d8d4dc4a43a1a391ea74418418" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "631566d8d4dc4a43a1a391ea74418418" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 45397 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:55:08.352515 23259 tablet_service.cc:1514] Processing DeleteTablet for tablet 72b01a904eab46e7865356ab5358acb6 with delete_type TABLET_DATA_TOMBSTONED (TS 22b61cbb0f2148db8f62df6a32215c8d not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:59848
I20240502 01:55:08.353387 24538 tablet_replica.cc:330] T 72b01a904eab46e7865356ab5358acb6 P 22b61cbb0f2148db8f62df6a32215c8d: stopping tablet replica
I20240502 01:55:08.353801 24538 raft_consensus.cc:2229] T 72b01a904eab46e7865356ab5358acb6 P 22b61cbb0f2148db8f62df6a32215c8d [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:08.354061 24538 raft_consensus.cc:2258] T 72b01a904eab46e7865356ab5358acb6 P 22b61cbb0f2148db8f62df6a32215c8d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:08.355432 24538 ts_tablet_manager.cc:1875] T 72b01a904eab46e7865356ab5358acb6 P 22b61cbb0f2148db8f62df6a32215c8d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:08.360502 24538 ts_tablet_manager.cc:1888] T 72b01a904eab46e7865356ab5358acb6 P 22b61cbb0f2148db8f62df6a32215c8d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240502 01:55:08.360687 24538 log.cc:1196] T 72b01a904eab46e7865356ab5358acb6 P 22b61cbb0f2148db8f62df6a32215c8d: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/wal/wals/72b01a904eab46e7865356ab5358acb6
I20240502 01:55:08.361688 22993 catalog_manager.cc:4830] TS 22b61cbb0f2148db8f62df6a32215c8d (127.9.176.194:35853): tablet 72b01a904eab46e7865356ab5358acb6 (table rebalance_test_table_3 [id=88b8e1b8224046a2bcff6f69c6233906]) successfully deleted
I20240502 01:55:08.393818 24007 consensus_queue.cc:237] T 6213c1863b2048c1a281634dd2693c3e P 631566d8d4dc4a43a1a391ea74418418 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "631566d8d4dc4a43a1a391ea74418418" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 45397 } attrs { promote: false } }
I20240502 01:55:08.395015 24536 raft_consensus.cc:2932] T 6213c1863b2048c1a281634dd2693c3e P 631566d8d4dc4a43a1a391ea74418418 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 22b61cbb0f2148db8f62df6a32215c8d (127.9.176.194) evicted. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "631566d8d4dc4a43a1a391ea74418418" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 45397 } attrs { promote: false } } }
I20240502 01:55:08.399240 22992 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6213c1863b2048c1a281634dd2693c3e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240502 01:55:08.400903 23148 raft_consensus.cc:1261] T 49eddfad6f6e4247a83e82964e8a9470 P 29f84c922f9f47f8bd7c360af478d359 [term 2 FOLLOWER]: Refusing update from remote peer 218f63e888ff4c6394af93014b59d65f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 01:55:08.401665 23007 catalog_manager.cc:5484] T 6213c1863b2048c1a281634dd2693c3e P 631566d8d4dc4a43a1a391ea74418418 reported cstate change: config changed from index 3 to 5, VOTER 22b61cbb0f2148db8f62df6a32215c8d (127.9.176.194) evicted. New cstate: current_term: 2 leader_uuid: "631566d8d4dc4a43a1a391ea74418418" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "631566d8d4dc4a43a1a391ea74418418" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 45397 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:55:08.401715 24267 consensus_queue.cc:1035] T 49eddfad6f6e4247a83e82964e8a9470 P 218f63e888ff4c6394af93014b59d65f [LEADER]: Connected to new peer: Peer: permanent_uuid: "29f84c922f9f47f8bd7c360af478d359" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 42839 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:08.406013 23259 tablet_service.cc:1514] Processing DeleteTablet for tablet 6213c1863b2048c1a281634dd2693c3e with delete_type TABLET_DATA_TOMBSTONED (TS 22b61cbb0f2148db8f62df6a32215c8d not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:59848
I20240502 01:55:08.406633 24538 tablet_replica.cc:330] T 6213c1863b2048c1a281634dd2693c3e P 22b61cbb0f2148db8f62df6a32215c8d: stopping tablet replica
I20240502 01:55:08.406845 24538 raft_consensus.cc:2229] T 6213c1863b2048c1a281634dd2693c3e P 22b61cbb0f2148db8f62df6a32215c8d [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:08.407099 24538 raft_consensus.cc:2258] T 6213c1863b2048c1a281634dd2693c3e P 22b61cbb0f2148db8f62df6a32215c8d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:08.409189 24538 ts_tablet_manager.cc:1875] T 6213c1863b2048c1a281634dd2693c3e P 22b61cbb0f2148db8f62df6a32215c8d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:08.417806 24538 ts_tablet_manager.cc:1888] T 6213c1863b2048c1a281634dd2693c3e P 22b61cbb0f2148db8f62df6a32215c8d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240502 01:55:08.417999 24538 log.cc:1196] T 6213c1863b2048c1a281634dd2693c3e P 22b61cbb0f2148db8f62df6a32215c8d: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/wal/wals/6213c1863b2048c1a281634dd2693c3e
I20240502 01:55:08.419157 22993 catalog_manager.cc:4830] TS 22b61cbb0f2148db8f62df6a32215c8d (127.9.176.194:35853): tablet 6213c1863b2048c1a281634dd2693c3e (table rebalance_test_table_1 [id=80ac4d7f091449898caeebdad7d14f29]) successfully deleted
I20240502 01:55:08.445607 23866 consensus_queue.cc:237] T 49eddfad6f6e4247a83e82964e8a9470 P 218f63e888ff4c6394af93014b59d65f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "218f63e888ff4c6394af93014b59d65f" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 34605 } attrs { promote: false } }
I20240502 01:55:08.446645 24551 raft_consensus.cc:2932] T 49eddfad6f6e4247a83e82964e8a9470 P 218f63e888ff4c6394af93014b59d65f [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 29f84c922f9f47f8bd7c360af478d359 (127.9.176.193) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "218f63e888ff4c6394af93014b59d65f" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 34605 } attrs { promote: false } } }
I20240502 01:55:08.449821 22993 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 49eddfad6f6e4247a83e82964e8a9470 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240502 01:55:08.451295 23007 catalog_manager.cc:5484] T 49eddfad6f6e4247a83e82964e8a9470 P 218f63e888ff4c6394af93014b59d65f reported cstate change: config changed from index 4 to 6, VOTER 29f84c922f9f47f8bd7c360af478d359 (127.9.176.193) evicted. New cstate: current_term: 2 leader_uuid: "218f63e888ff4c6394af93014b59d65f" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "218f63e888ff4c6394af93014b59d65f" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 34605 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:55:08.455181 23128 tablet_service.cc:1514] Processing DeleteTablet for tablet 49eddfad6f6e4247a83e82964e8a9470 with delete_type TABLET_DATA_TOMBSTONED (TS 29f84c922f9f47f8bd7c360af478d359 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52566
I20240502 01:55:08.456007 24571 tablet_replica.cc:330] T 49eddfad6f6e4247a83e82964e8a9470 P 29f84c922f9f47f8bd7c360af478d359: stopping tablet replica
I20240502 01:55:08.456470 24571 raft_consensus.cc:2229] T 49eddfad6f6e4247a83e82964e8a9470 P 29f84c922f9f47f8bd7c360af478d359 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:08.456820 24571 raft_consensus.cc:2258] T 49eddfad6f6e4247a83e82964e8a9470 P 29f84c922f9f47f8bd7c360af478d359 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:08.458465 24571 ts_tablet_manager.cc:1875] T 49eddfad6f6e4247a83e82964e8a9470 P 29f84c922f9f47f8bd7c360af478d359: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:08.465413 24571 ts_tablet_manager.cc:1888] T 49eddfad6f6e4247a83e82964e8a9470 P 29f84c922f9f47f8bd7c360af478d359: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240502 01:55:08.465663 24571 log.cc:1196] T 49eddfad6f6e4247a83e82964e8a9470 P 29f84c922f9f47f8bd7c360af478d359: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/wal/wals/49eddfad6f6e4247a83e82964e8a9470
I20240502 01:55:08.466517 22993 catalog_manager.cc:4830] TS 29f84c922f9f47f8bd7c360af478d359 (127.9.176.193:42839): tablet 49eddfad6f6e4247a83e82964e8a9470 (table rebalance_test_table_2 [id=6256f1b0e93c4aca801bfb7d7031a5af]) successfully deleted
I20240502 01:55:08.475672 24007 consensus_queue.cc:237] T 209d6a3dc32f43b1b4d536e02bd0ec73 P 631566d8d4dc4a43a1a391ea74418418 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "631566d8d4dc4a43a1a391ea74418418" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 45397 } attrs { promote: false } }
I20240502 01:55:08.476573 24536 raft_consensus.cc:2932] T 209d6a3dc32f43b1b4d536e02bd0ec73 P 631566d8d4dc4a43a1a391ea74418418 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 29f84c922f9f47f8bd7c360af478d359 (127.9.176.193) evicted. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "631566d8d4dc4a43a1a391ea74418418" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 45397 } attrs { promote: false } } }
I20240502 01:55:08.479241 22992 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 209d6a3dc32f43b1b4d536e02bd0ec73 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240502 01:55:08.480690 23007 catalog_manager.cc:5484] T 209d6a3dc32f43b1b4d536e02bd0ec73 P 631566d8d4dc4a43a1a391ea74418418 reported cstate change: config changed from index 3 to 5, VOTER 29f84c922f9f47f8bd7c360af478d359 (127.9.176.193) evicted. New cstate: current_term: 2 leader_uuid: "631566d8d4dc4a43a1a391ea74418418" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "631566d8d4dc4a43a1a391ea74418418" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 45397 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:55:08.484333 23128 tablet_service.cc:1514] Processing DeleteTablet for tablet 209d6a3dc32f43b1b4d536e02bd0ec73 with delete_type TABLET_DATA_TOMBSTONED (TS 29f84c922f9f47f8bd7c360af478d359 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:52566
I20240502 01:55:08.484695 24571 tablet_replica.cc:330] T 209d6a3dc32f43b1b4d536e02bd0ec73 P 29f84c922f9f47f8bd7c360af478d359: stopping tablet replica
I20240502 01:55:08.484869 24571 raft_consensus.cc:2229] T 209d6a3dc32f43b1b4d536e02bd0ec73 P 29f84c922f9f47f8bd7c360af478d359 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:08.485038 24571 raft_consensus.cc:2258] T 209d6a3dc32f43b1b4d536e02bd0ec73 P 29f84c922f9f47f8bd7c360af478d359 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:08.485736 24571 ts_tablet_manager.cc:1875] T 209d6a3dc32f43b1b4d536e02bd0ec73 P 29f84c922f9f47f8bd7c360af478d359: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:08.490372 24571 ts_tablet_manager.cc:1888] T 209d6a3dc32f43b1b4d536e02bd0ec73 P 29f84c922f9f47f8bd7c360af478d359: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240502 01:55:08.490497 24571 log.cc:1196] T 209d6a3dc32f43b1b4d536e02bd0ec73 P 29f84c922f9f47f8bd7c360af478d359: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/wal/wals/209d6a3dc32f43b1b4d536e02bd0ec73
I20240502 01:55:08.491371 22993 catalog_manager.cc:4830] TS 29f84c922f9f47f8bd7c360af478d359 (127.9.176.193:42839): tablet 209d6a3dc32f43b1b4d536e02bd0ec73 (table rebalance_test_table_0 [id=a8cec354879c415095f48afc7deafeba]) successfully deleted
I20240502 01:55:08.643687 23148 raft_consensus.cc:1261] T 1f4cf78efc5f479c875536136d17fdb0 P 29f84c922f9f47f8bd7c360af478d359 [term 2 FOLLOWER]: Refusing update from remote peer 218f63e888ff4c6394af93014b59d65f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 01:55:08.644327 24551 consensus_queue.cc:1035] T 1f4cf78efc5f479c875536136d17fdb0 P 218f63e888ff4c6394af93014b59d65f [LEADER]: Connected to new peer: Peer: permanent_uuid: "29f84c922f9f47f8bd7c360af478d359" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 42839 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:08.690819 23866 consensus_queue.cc:237] T 1f4cf78efc5f479c875536136d17fdb0 P 218f63e888ff4c6394af93014b59d65f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "218f63e888ff4c6394af93014b59d65f" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 34605 } attrs { promote: false } }
I20240502 01:55:08.691845 24551 raft_consensus.cc:2932] T 1f4cf78efc5f479c875536136d17fdb0 P 218f63e888ff4c6394af93014b59d65f [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 29f84c922f9f47f8bd7c360af478d359 (127.9.176.193) evicted. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "218f63e888ff4c6394af93014b59d65f" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 34605 } attrs { promote: false } } }
I20240502 01:55:08.694880 22993 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1f4cf78efc5f479c875536136d17fdb0 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240502 01:55:08.696238 23005 catalog_manager.cc:5484] T 1f4cf78efc5f479c875536136d17fdb0 P 218f63e888ff4c6394af93014b59d65f reported cstate change: config changed from index 3 to 5, VOTER 29f84c922f9f47f8bd7c360af478d359 (127.9.176.193) evicted. New cstate: current_term: 2 leader_uuid: "218f63e888ff4c6394af93014b59d65f" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "218f63e888ff4c6394af93014b59d65f" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 34605 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:55:08.700330 23128 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f4cf78efc5f479c875536136d17fdb0 with delete_type TABLET_DATA_TOMBSTONED (TS 29f84c922f9f47f8bd7c360af478d359 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:52566
I20240502 01:55:08.700702 24571 tablet_replica.cc:330] T 1f4cf78efc5f479c875536136d17fdb0 P 29f84c922f9f47f8bd7c360af478d359: stopping tablet replica
I20240502 01:55:08.700902 24571 raft_consensus.cc:2229] T 1f4cf78efc5f479c875536136d17fdb0 P 29f84c922f9f47f8bd7c360af478d359 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:08.701176 24571 raft_consensus.cc:2258] T 1f4cf78efc5f479c875536136d17fdb0 P 29f84c922f9f47f8bd7c360af478d359 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:08.702001 24571 ts_tablet_manager.cc:1875] T 1f4cf78efc5f479c875536136d17fdb0 P 29f84c922f9f47f8bd7c360af478d359: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:08.708142 24571 ts_tablet_manager.cc:1888] T 1f4cf78efc5f479c875536136d17fdb0 P 29f84c922f9f47f8bd7c360af478d359: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240502 01:55:08.708297 24571 log.cc:1196] T 1f4cf78efc5f479c875536136d17fdb0 P 29f84c922f9f47f8bd7c360af478d359: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/wal/wals/1f4cf78efc5f479c875536136d17fdb0
I20240502 01:55:08.709085 22993 catalog_manager.cc:4830] TS 29f84c922f9f47f8bd7c360af478d359 (127.9.176.193:42839): tablet 1f4cf78efc5f479c875536136d17fdb0 (table rebalance_test_table_1 [id=80ac4d7f091449898caeebdad7d14f29]) successfully deleted
I20240502 01:55:08.826078 23845 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:55:08.838871 23988 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:55:08.848466 23259 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:55:08.851048 23128 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:55:09.004945 23845 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:55:09.005658 23259 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:55:09.008234 23988 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:55:09.010481 23128 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:55:09.308171 23259 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:55:09.308212 23845 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:55:09.317262 23128 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:55:09.319983 23988 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
Master Summary
               UUID               |       Address       | Status
----------------------------------+---------------------+---------
 4378e3dea314412ab6ba0f0dc7a84669 | 127.9.176.254:44139 | HEALTHY

Unusual flags for Master:
                   Flag                   |                                                                                               Value                                                                                               |         Tags          |         Master
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
 ipki_ca_key_size                         | 768                                                                                                                                                                                               | experimental          | all 1 server(s) checked
 ipki_server_key_size                     | 768                                                                                                                                                                                               | experimental          | all 1 server(s) checked
 never_fsync                              | true                                                                                                                                                                                              | unsafe,advanced       | all 1 server(s) checked
 openssl_security_level_override          | 0                                                                                                                                                                                                 | unsafe,hidden         | all 1 server(s) checked
 raft_prepare_replacement_before_eviction | true                                                                                                                                                                                              | experimental,advanced | all 1 server(s) checked
 rpc_reuseport                            | true                                                                                                                                                                                              | experimental          | all 1 server(s) checked
 rpc_server_allow_ephemeral_ports         | true                                                                                                                                                                                              | unsafe                | all 1 server(s) checked
 server_dump_info_format                  | pb                                                                                                                                                                                                | hidden                | all 1 server(s) checked
 server_dump_info_path                    | /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden                | all 1 server(s) checked
 tsk_num_rsa_bits                         | 512                                                                                                                                                                                               | experimental          | all 1 server(s) checked

Flags of checked categories for Master:
        Flag         |        Value        |         Master
---------------------+---------------------+-------------------------
 builtin_ntp_servers | 127.9.176.212:43017 | all 1 server(s) checked
 time_source         | builtin             | all 1 server(s) checked

Tablet Server Summary
               UUID               |       Address       | Status  | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
 218f63e888ff4c6394af93014b59d65f | 127.9.176.195:34605 | HEALTHY | <none>   |       3        |       0
 22b61cbb0f2148db8f62df6a32215c8d | 127.9.176.194:35853 | HEALTHY | <none>   |       4        |       0
 29f84c922f9f47f8bd7c360af478d359 | 127.9.176.193:42839 | HEALTHY | <none>   |       3        |       0
 631566d8d4dc4a43a1a391ea74418418 | 127.9.176.196:45397 | HEALTHY | <none>   |       3        |       0

Tablet Server Location Summary
 Location |  Count
----------+---------
 <none>   |       4

Unusual flags for Tablet Server:
                   Flag                   |                                                                                             Value                                                                                             |         Tags          |      Tablet Server
------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
 ipki_server_key_size                     | 768                                                                                                                                                                                           | experimental          | all 4 server(s) checked
 local_ip_for_outbound_sockets            | 127.9.176.193                                                                                                                                                                                 | experimental          | 127.9.176.193:42839
 local_ip_for_outbound_sockets            | 127.9.176.194                                                                                                                                                                                 | experimental          | 127.9.176.194:35853
 local_ip_for_outbound_sockets            | 127.9.176.195                                                                                                                                                                                 | experimental          | 127.9.176.195:34605
 local_ip_for_outbound_sockets            | 127.9.176.196                                                                                                                                                                                 | experimental          | 127.9.176.196:45397
 never_fsync                              | true                                                                                                                                                                                          | unsafe,advanced       | all 4 server(s) checked
 openssl_security_level_override          | 0                                                                                                                                                                                             | unsafe,hidden         | all 4 server(s) checked
 raft_prepare_replacement_before_eviction | true                                                                                                                                                                                          | experimental,advanced | all 4 server(s) checked
 rpc_server_allow_ephemeral_ports         | true                                                                                                                                                                                          | unsafe                | all 4 server(s) checked
 server_dump_info_format                  | pb                                                                                                                                                                                            | hidden                | all 4 server(s) checked
 server_dump_info_path                    | /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden                | 127.9.176.193:42839
 server_dump_info_path                    | /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden                | 127.9.176.194:35853
 server_dump_info_path                    | /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden                | 127.9.176.195:34605
 server_dump_info_path                    | /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden                | 127.9.176.196:45397

Flags of checked categories for Tablet Server:
        Flag         |        Value        |      Tablet Server
---------------------+---------------------+-------------------------
 builtin_ntp_servers | 127.9.176.212:43017 | all 4 server(s) checked
 time_source         | builtin             | all 4 server(s) checked

Version Summary
     Version     |         Servers
-----------------+-------------------------
 1.18.0-SNAPSHOT | all 5 server(s) checked

Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
          Name          | RF | Status  | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
 TestTable              | 1  | HEALTHY | 1             | 1       | 0          | 0                | 0
 rebalance_test_table_0 | 1  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_1 | 1  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_2 | 1  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_3 | 1  | HEALTHY | 3             | 3       | 0          | 0                | 0

Tablet Replica Count Summary
   Statistic    | Replica Count
----------------+---------------
 Minimum        | 3
 First Quartile | 3
 Median         | 3
 Third Quartile | 4
 Maximum        | 4

Total Count Summary
                | Total Count
----------------+-------------
 Masters        | 1
 Tablet Servers | 4
 Tables         | 5
 Tablets        | 13
 Replicas       | 13

==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set

OK
I20240502 01:55:09.374646  9923 log_verifier.cc:126] Checking tablet 0e2012468bb94a30a1d62cb58a98fc8c
I20240502 01:55:09.420709  9923 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 0e2012468bb94a30a1d62cb58a98fc8c
I20240502 01:55:09.420810  9923 log_verifier.cc:126] Checking tablet 1f4cf78efc5f479c875536136d17fdb0
I20240502 01:55:09.429201  9923 log_verifier.cc:177] Verified matching terms for 5 ops in tablet 1f4cf78efc5f479c875536136d17fdb0
I20240502 01:55:09.429299  9923 log_verifier.cc:126] Checking tablet 209d6a3dc32f43b1b4d536e02bd0ec73
I20240502 01:55:09.437912  9923 log_verifier.cc:177] Verified matching terms for 5 ops in tablet 209d6a3dc32f43b1b4d536e02bd0ec73
I20240502 01:55:09.438026  9923 log_verifier.cc:126] Checking tablet 23dff2c6cd704ce6a8853d393dd97119
I20240502 01:55:09.446431  9923 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 23dff2c6cd704ce6a8853d393dd97119
I20240502 01:55:09.446538  9923 log_verifier.cc:126] Checking tablet 310930a2b5a7416d82706f03e06a5daa
I20240502 01:55:09.454682  9923 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 310930a2b5a7416d82706f03e06a5daa
I20240502 01:55:09.454782  9923 log_verifier.cc:126] Checking tablet 33acbbf5980c41adbc6b2ac5d5ddb0ae
I20240502 01:55:09.462540  9923 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 33acbbf5980c41adbc6b2ac5d5ddb0ae
I20240502 01:55:09.462643  9923 log_verifier.cc:126] Checking tablet 49eddfad6f6e4247a83e82964e8a9470
I20240502 01:55:09.471323  9923 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 49eddfad6f6e4247a83e82964e8a9470
I20240502 01:55:09.471429  9923 log_verifier.cc:126] Checking tablet 5c23236383d646d29f39a5c18938fd24
I20240502 01:55:09.479226  9923 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 5c23236383d646d29f39a5c18938fd24
I20240502 01:55:09.479319  9923 log_verifier.cc:126] Checking tablet 6213c1863b2048c1a281634dd2693c3e
I20240502 01:55:09.488224  9923 log_verifier.cc:177] Verified matching terms for 5 ops in tablet 6213c1863b2048c1a281634dd2693c3e
I20240502 01:55:09.488324  9923 log_verifier.cc:126] Checking tablet 6341db525c364256bc0450f381207ad3
I20240502 01:55:09.496420  9923 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 6341db525c364256bc0450f381207ad3
I20240502 01:55:09.496520  9923 log_verifier.cc:126] Checking tablet 72b01a904eab46e7865356ab5358acb6
I20240502 01:55:09.505017  9923 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 72b01a904eab46e7865356ab5358acb6
I20240502 01:55:09.505146  9923 log_verifier.cc:126] Checking tablet b9422d7f0ab644979df332552c1e9d14
I20240502 01:55:09.512754  9923 log_verifier.cc:177] Verified matching terms for 1 ops in tablet b9422d7f0ab644979df332552c1e9d14
I20240502 01:55:09.512845  9923 log_verifier.cc:126] Checking tablet d6ef9d418c534f8ab088edcd3b372856
I20240502 01:55:09.520409  9923 log_verifier.cc:177] Verified matching terms for 1 ops in tablet d6ef9d418c534f8ab088edcd3b372856
I20240502 01:55:09.521302  9923 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu pid 23066
I20240502 01:55:09.522747 23187 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:55:09.650305  9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu with pid 23066
I20240502 01:55:09.673085  9923 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu pid 23197
I20240502 01:55:09.674166 23319 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:55:09.790745  9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu with pid 23197
I20240502 01:55:09.813907  9923 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu pid 23782
I20240502 01:55:09.820233 23907 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:55:09.928078  9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu with pid 23782
I20240502 01:55:09.948688  9923 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu pid 23921
I20240502 01:55:09.954675 24047 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:55:10.061873  9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu with pid 23921
I20240502 01:55:10.082264  9923 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu pid 22976
I20240502 01:55:10.083434 23037 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:55:10.218499  9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu with pid 22976
2024-05-02T01:55:10Z chronyd exiting
[       OK ] RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove/1 (18445 ms)
[----------] 1 test from RebalancerAndSingleReplicaTablets (18445 ms total)

[----------] 2 tests from LocationAwareRebalancingParamTest
[ RUN      ] LocationAwareRebalancingParamTest.Rebalance/idx3_rf3_t3_l_3_3
I20240502 01:55:10.279357  9923 test_util.cc:268] Using random seed: -458793296
I20240502 01:55:10.281214  9923 ts_itest-base.cc:112] Starting cluster with:
I20240502 01:55:10.281297  9923 ts_itest-base.cc:113] --------------
I20240502 01:55:10.281333  9923 ts_itest-base.cc:114] 6 tablet servers
I20240502 01:55:10.281353  9923 ts_itest-base.cc:115] 3 replicas per TS
I20240502 01:55:10.281379  9923 ts_itest-base.cc:116] --------------
2024-05-02T01:55:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T01:55:10Z Disabled control of system clock
I20240502 01:55:10.339121  9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
/tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:38303
--webserver_interface=127.9.176.254
--webserver_port=0
--builtin_ntp_servers=127.9.176.212:33577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.176.254:38303
--location_mapping_cmd=/tmp/dist-test-taskJZxTxO/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/location-assignment.state --map /A:3 --map /B:3
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false
--master_client_location_assignment_enabled=false
--allow_unsafe_replication_factor with env {}
W20240502 01:55:10.702930 24696 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240502 01:55:10.703374 24696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:55:10.703483 24696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:55:10.703631 24696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:55:10.711635 24696 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:55:10.711810 24696 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 01:55:10.711917 24696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:55:10.711987 24696 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 01:55:10.712080 24696 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 01:55:10.722056 24696 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:33577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--location_mapping_cmd=/tmp/dist-test-taskJZxTxO/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/location-assignment.state --map /A:3 --map /B:3
--ipki_ca_key_size=768
--master_addresses=127.9.176.254:38303
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:38303
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.9.176.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true

Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:55:10.723610 24696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:55:10.725466 24696 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:55:10.737366 24701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:10.737850 24702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:10.738276 24704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:55:10.739085 24696 server_base.cc:1034] running on GCE node
I20240502 01:55:10.740017 24696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:55:10.741636 24696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:55:10.742941 24696 hybrid_clock.cc:648] HybridClock initialized: now 1714614910742857 us; error 56 us; skew 500 ppm
I20240502 01:55:10.743330 24696 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:55:10.746610 24696 webserver.cc:460] Webserver started at http://127.9.176.254:43247/ using document root <none> and password file <none>
I20240502 01:55:10.747476 24696 fs_manager.cc:359] Metadata directory not provided
I20240502 01:55:10.747664 24696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:55:10.748015 24696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:55:10.750885 24696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "6486e967e6ee4994a2fad0f2c0c7c8f4"
format_stamp: "Formatted at 2024-05-02 01:55:10 on dist-test-slave-j3hh"
I20240502 01:55:10.751813 24696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "6486e967e6ee4994a2fad0f2c0c7c8f4"
format_stamp: "Formatted at 2024-05-02 01:55:10 on dist-test-slave-j3hh"
I20240502 01:55:10.757570 24696 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.002s	sys 0.002s
I20240502 01:55:10.762634 24710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:55:10.765450 24696 fs_manager.cc:725] Time spent opening block manager: real 0.005s	user 0.002s	sys 0.000s
I20240502 01:55:10.765682 24696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/wal
uuid: "6486e967e6ee4994a2fad0f2c0c7c8f4"
format_stamp: "Formatted at 2024-05-02 01:55:10 on dist-test-slave-j3hh"
I20240502 01:55:10.765937 24696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:55:10.789098 24696 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:55:10.789999 24696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:55:10.790318 24696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:55:10.810359 24696 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.254:38303
I20240502 01:55:10.810379 24761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.254:38303 every 8 connection(s)
I20240502 01:55:10.811919 24696 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 01:55:10.815378  9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu as pid 24696
I20240502 01:55:10.815654  9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 01:55:10.816318 24762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:10.829377 24762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6486e967e6ee4994a2fad0f2c0c7c8f4: Bootstrap starting.
I20240502 01:55:10.832962 24762 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6486e967e6ee4994a2fad0f2c0c7c8f4: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:10.834187 24762 log.cc:824] T 00000000000000000000000000000000 P 6486e967e6ee4994a2fad0f2c0c7c8f4: Log is configured to *not* fsync() on all Append() calls
I20240502 01:55:10.837579 24762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6486e967e6ee4994a2fad0f2c0c7c8f4: No bootstrap required, opened a new log
I20240502 01:55:10.844656 24762 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6486e967e6ee4994a2fad0f2c0c7c8f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6486e967e6ee4994a2fad0f2c0c7c8f4" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 38303 } }
I20240502 01:55:10.845093 24762 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6486e967e6ee4994a2fad0f2c0c7c8f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:10.845208 24762 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6486e967e6ee4994a2fad0f2c0c7c8f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6486e967e6ee4994a2fad0f2c0c7c8f4, State: Initialized, Role: FOLLOWER
I20240502 01:55:10.845909 24762 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6486e967e6ee4994a2fad0f2c0c7c8f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6486e967e6ee4994a2fad0f2c0c7c8f4" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 38303 } }
I20240502 01:55:10.846134 24762 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6486e967e6ee4994a2fad0f2c0c7c8f4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:55:10.846285 24762 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6486e967e6ee4994a2fad0f2c0c7c8f4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:55:10.846462 24762 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6486e967e6ee4994a2fad0f2c0c7c8f4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:10.849121 24762 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6486e967e6ee4994a2fad0f2c0c7c8f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6486e967e6ee4994a2fad0f2c0c7c8f4" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 38303 } }
I20240502 01:55:10.849759 24762 leader_election.cc:302] T 00000000000000000000000000000000 P 6486e967e6ee4994a2fad0f2c0c7c8f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6486e967e6ee4994a2fad0f2c0c7c8f4; no voters: 
I20240502 01:55:10.850417 24762 leader_election.cc:288] T 00000000000000000000000000000000 P 6486e967e6ee4994a2fad0f2c0c7c8f4 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240502 01:55:10.850567 24767 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6486e967e6ee4994a2fad0f2c0c7c8f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:55:10.851325 24767 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6486e967e6ee4994a2fad0f2c0c7c8f4 [term 1 LEADER]: Becoming Leader. State: Replica: 6486e967e6ee4994a2fad0f2c0c7c8f4, State: Running, Role: LEADER
I20240502 01:55:10.851946 24767 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6486e967e6ee4994a2fad0f2c0c7c8f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6486e967e6ee4994a2fad0f2c0c7c8f4" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 38303 } }
I20240502 01:55:10.852838 24762 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6486e967e6ee4994a2fad0f2c0c7c8f4 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 01:55:10.855078 24768 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6486e967e6ee4994a2fad0f2c0c7c8f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6486e967e6ee4994a2fad0f2c0c7c8f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6486e967e6ee4994a2fad0f2c0c7c8f4" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 38303 } } }
I20240502 01:55:10.855052 24769 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6486e967e6ee4994a2fad0f2c0c7c8f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6486e967e6ee4994a2fad0f2c0c7c8f4. Latest consensus state: current_term: 1 leader_uuid: "6486e967e6ee4994a2fad0f2c0c7c8f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6486e967e6ee4994a2fad0f2c0c7c8f4" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 38303 } } }
I20240502 01:55:10.855531 24769 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6486e967e6ee4994a2fad0f2c0c7c8f4 [sys.catalog]: This master's current role is: LEADER
I20240502 01:55:10.855677 24768 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6486e967e6ee4994a2fad0f2c0c7c8f4 [sys.catalog]: This master's current role is: LEADER
I20240502 01:55:10.862454 24772 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 01:55:10.868405 24772 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 01:55:10.877794 24772 catalog_manager.cc:1347] Generated new cluster ID: c86c957f826c4251bb5f6ed095535b31
I20240502 01:55:10.877935 24772 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 01:55:10.909936 24772 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 01:55:10.911902 24772 catalog_manager.cc:1504] Loading token signing keys...
I20240502 01:55:10.930058 24772 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 6486e967e6ee4994a2fad0f2c0c7c8f4: Generated new TSK 0
I20240502 01:55:10.930981 24772 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 01:55:10.942792  9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
/tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.193:0
--local_ip_for_outbound_sockets=127.9.176.193
--webserver_interface=127.9.176.193
--webserver_port=0
--tserver_master_addrs=127.9.176.254:38303
--builtin_ntp_servers=127.9.176.212:33577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 01:55:11.312088 24787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:55:11.312418 24787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:55:11.312577 24787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:55:11.320513 24787 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:55:11.320655 24787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:55:11.320835 24787 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.193
I20240502 01:55:11.330438 24787 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:33577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.9.176.193
--webserver_port=0
--tserver_master_addrs=127.9.176.254:38303
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.193
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:55:11.331883 24787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:55:11.333711 24787 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:55:11.346568 24792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:11.346992 24795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:11.347397 24793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:55:11.348290 24787 server_base.cc:1034] running on GCE node
I20240502 01:55:11.348963 24787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:55:11.350005 24787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:55:11.351226 24787 hybrid_clock.cc:648] HybridClock initialized: now 1714614911351151 us; error 65 us; skew 500 ppm
I20240502 01:55:11.351608 24787 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:55:11.354130 24787 webserver.cc:460] Webserver started at http://127.9.176.193:38117/ using document root <none> and password file <none>
I20240502 01:55:11.354894 24787 fs_manager.cc:359] Metadata directory not provided
I20240502 01:55:11.355011 24787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:55:11.355355 24787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:55:11.357937 24787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "96e5c2fd5d624c2da735e72b387a12d9"
format_stamp: "Formatted at 2024-05-02 01:55:11 on dist-test-slave-j3hh"
I20240502 01:55:11.358728 24787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "96e5c2fd5d624c2da735e72b387a12d9"
format_stamp: "Formatted at 2024-05-02 01:55:11 on dist-test-slave-j3hh"
I20240502 01:55:11.364432 24787 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.006s	sys 0.001s
I20240502 01:55:11.368583 24801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:55:11.370383 24787 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.004s	sys 0.000s
I20240502 01:55:11.370595 24787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "96e5c2fd5d624c2da735e72b387a12d9"
format_stamp: "Formatted at 2024-05-02 01:55:11 on dist-test-slave-j3hh"
I20240502 01:55:11.370828 24787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:55:11.404213 24787 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:55:11.405175 24787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:55:11.405467 24787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:55:11.406529 24787 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:55:11.408239 24787 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:55:11.408349 24787 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:55:11.408478 24787 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:55:11.408564 24787 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:55:11.443199 24787 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.193:32783
I20240502 01:55:11.443279 24913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.193:32783 every 8 connection(s)
I20240502 01:55:11.444664 24787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 01:55:11.452791  9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu as pid 24787
I20240502 01:55:11.453028  9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 01:55:11.456946  9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
/tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.194:0
--local_ip_for_outbound_sockets=127.9.176.194
--webserver_interface=127.9.176.194
--webserver_port=0
--tserver_master_addrs=127.9.176.254:38303
--builtin_ntp_servers=127.9.176.212:33577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:55:11.458595 24914 heartbeater.cc:347] Connected to a master server at 127.9.176.254:38303
I20240502 01:55:11.458956 24914 heartbeater.cc:464] Registering TS with master...
I20240502 01:55:11.460089 24914 heartbeater.cc:511] Master 127.9.176.254:38303 requested a full tablet report, sending...
I20240502 01:55:11.521788 24727 ts_manager.cc:194] Registered new tserver with Master: 96e5c2fd5d624c2da735e72b387a12d9 (127.9.176.193:32783)
I20240502 01:55:11.523638 24727 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.176.193:59801
W20240502 01:55:11.826006 24918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:55:11.826323 24918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:55:11.826471 24918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:55:11.834590 24918 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:55:11.834736 24918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:55:11.834910 24918 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.194
I20240502 01:55:11.844573 24918 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:33577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.9.176.194
--webserver_port=0
--tserver_master_addrs=127.9.176.254:38303
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.194
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:55:11.845970 24918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:55:11.847930 24918 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:55:11.860980 24924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:11.861497 24925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:11.861999 24927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:55:11.862694 24918 server_base.cc:1034] running on GCE node
I20240502 01:55:11.863442 24918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:55:11.864625 24918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:55:11.865855 24918 hybrid_clock.cc:648] HybridClock initialized: now 1714614911865798 us; error 53 us; skew 500 ppm
I20240502 01:55:11.866215 24918 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:55:11.868829 24918 webserver.cc:460] Webserver started at http://127.9.176.194:35251/ using document root <none> and password file <none>
I20240502 01:55:11.869640 24918 fs_manager.cc:359] Metadata directory not provided
I20240502 01:55:11.869772 24918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:55:11.870112 24918 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:55:11.873051 24918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "75bd3f0e8d7c4e8aa92a353301afeb21"
format_stamp: "Formatted at 2024-05-02 01:55:11 on dist-test-slave-j3hh"
I20240502 01:55:11.873963 24918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "75bd3f0e8d7c4e8aa92a353301afeb21"
format_stamp: "Formatted at 2024-05-02 01:55:11 on dist-test-slave-j3hh"
I20240502 01:55:11.879931 24918 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.007s	sys 0.000s
I20240502 01:55:11.883823 24933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:55:11.885315 24918 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20240502 01:55:11.885535 24918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "75bd3f0e8d7c4e8aa92a353301afeb21"
format_stamp: "Formatted at 2024-05-02 01:55:11 on dist-test-slave-j3hh"
I20240502 01:55:11.885772 24918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:55:11.901011 24918 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:55:11.901964 24918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:55:11.902257 24918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:55:11.903301 24918 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:55:11.905005 24918 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:55:11.905103 24918 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:55:11.905241 24918 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:55:11.905313 24918 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:55:11.940296 24918 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.194:41625
I20240502 01:55:11.940352 25046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.194:41625 every 8 connection(s)
I20240502 01:55:11.941761 24918 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 01:55:11.950093  9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu as pid 24918
I20240502 01:55:11.950306  9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 01:55:11.954161  9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
/tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.195:0
--local_ip_for_outbound_sockets=127.9.176.195
--webserver_interface=127.9.176.195
--webserver_port=0
--tserver_master_addrs=127.9.176.254:38303
--builtin_ntp_servers=127.9.176.212:33577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:55:11.956454 25047 heartbeater.cc:347] Connected to a master server at 127.9.176.254:38303
I20240502 01:55:11.956784 25047 heartbeater.cc:464] Registering TS with master...
I20240502 01:55:11.957669 25047 heartbeater.cc:511] Master 127.9.176.254:38303 requested a full tablet report, sending...
I20240502 01:55:12.007946 24727 ts_manager.cc:194] Registered new tserver with Master: 75bd3f0e8d7c4e8aa92a353301afeb21 (127.9.176.194:41625)
I20240502 01:55:12.008967 24727 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.176.194:56133
W20240502 01:55:12.323336 25051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:55:12.323679 25051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:55:12.323810 25051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:55:12.331614 25051 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:55:12.331743 25051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:55:12.331909 25051 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.195
I20240502 01:55:12.341030 25051 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:33577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.9.176.195
--webserver_port=0
--tserver_master_addrs=127.9.176.254:38303
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.195
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:55:12.342381 25051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:55:12.344278 25051 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:55:12.356671 25058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:12.357056 25060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:12.357491 25057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:55:12.358268 25051 server_base.cc:1034] running on GCE node
I20240502 01:55:12.358875 25051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:55:12.359951 25051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:55:12.361155 25051 hybrid_clock.cc:648] HybridClock initialized: now 1714614912361111 us; error 38 us; skew 500 ppm
I20240502 01:55:12.361483 25051 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:55:12.364024 25051 webserver.cc:460] Webserver started at http://127.9.176.195:36267/ using document root <none> and password file <none>
I20240502 01:55:12.364849 25051 fs_manager.cc:359] Metadata directory not provided
I20240502 01:55:12.364988 25051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:55:12.365331 25051 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:55:12.368198 25051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "672a55e7d4874c5d99f3c5e3d9550f15"
format_stamp: "Formatted at 2024-05-02 01:55:12 on dist-test-slave-j3hh"
I20240502 01:55:12.369052 25051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "672a55e7d4874c5d99f3c5e3d9550f15"
format_stamp: "Formatted at 2024-05-02 01:55:12 on dist-test-slave-j3hh"
I20240502 01:55:12.374624 25051 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.004s	sys 0.002s
I20240502 01:55:12.378623 25066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:55:12.380267 25051 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240502 01:55:12.380466 25051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "672a55e7d4874c5d99f3c5e3d9550f15"
format_stamp: "Formatted at 2024-05-02 01:55:12 on dist-test-slave-j3hh"
I20240502 01:55:12.380707 25051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:55:12.410815 25051 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:55:12.411876 25051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:55:12.412175 25051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:55:12.413323 25051 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:55:12.414981 25051 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:55:12.415095 25051 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:55:12.415220 25051 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:55:12.415298 25051 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:55:12.448419 25051 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.195:46579
I20240502 01:55:12.448454 25178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.195:46579 every 8 connection(s)
I20240502 01:55:12.450045 25051 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 01:55:12.453541  9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu as pid 25051
I20240502 01:55:12.453718  9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 01:55:12.457461  9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
/tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.196:0
--local_ip_for_outbound_sockets=127.9.176.196
--webserver_interface=127.9.176.196
--webserver_port=0
--tserver_master_addrs=127.9.176.254:38303
--builtin_ntp_servers=127.9.176.212:33577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:55:12.465063 25179 heartbeater.cc:347] Connected to a master server at 127.9.176.254:38303
I20240502 01:55:12.465389 25179 heartbeater.cc:464] Registering TS with master...
I20240502 01:55:12.466271 25179 heartbeater.cc:511] Master 127.9.176.254:38303 requested a full tablet report, sending...
I20240502 01:55:12.527220 24914 heartbeater.cc:503] Master 127.9.176.254:38303 was elected leader, sending a full tablet report...
I20240502 01:55:12.534353 24727 ts_manager.cc:194] Registered new tserver with Master: 672a55e7d4874c5d99f3c5e3d9550f15 (127.9.176.195:46579)
I20240502 01:55:12.535322 24727 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.176.195:46847
W20240502 01:55:12.845399 25182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:55:12.845744 25182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:55:12.845844 25182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:55:12.854007 25182 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:55:12.854156 25182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:55:12.854317 25182 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.196
I20240502 01:55:12.864266 25182 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:33577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.9.176.196
--webserver_port=0
--tserver_master_addrs=127.9.176.254:38303
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.196
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:55:12.865712 25182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:55:12.867687 25182 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:55:12.881619 25188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:12.881970 25191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:12.882470 25189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:55:12.883004 25182 server_base.cc:1034] running on GCE node
I20240502 01:55:12.883755 25182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:55:12.884835 25182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:55:12.886124 25182 hybrid_clock.cc:648] HybridClock initialized: now 1714614912885993 us; error 117 us; skew 500 ppm
I20240502 01:55:12.886554 25182 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:55:12.893188 25182 webserver.cc:460] Webserver started at http://127.9.176.196:36127/ using document root <none> and password file <none>
I20240502 01:55:12.894189 25182 fs_manager.cc:359] Metadata directory not provided
I20240502 01:55:12.894371 25182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:55:12.894773 25182 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:55:12.897903 25182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "30921110704f47f88d68a7d9364d42cc"
format_stamp: "Formatted at 2024-05-02 01:55:12 on dist-test-slave-j3hh"
I20240502 01:55:12.898938 25182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "30921110704f47f88d68a7d9364d42cc"
format_stamp: "Formatted at 2024-05-02 01:55:12 on dist-test-slave-j3hh"
I20240502 01:55:12.905639 25182 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.009s	sys 0.000s
I20240502 01:55:12.910104 25197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:55:12.911895 25182 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.000s
I20240502 01:55:12.912123 25182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "30921110704f47f88d68a7d9364d42cc"
format_stamp: "Formatted at 2024-05-02 01:55:12 on dist-test-slave-j3hh"
I20240502 01:55:12.912384 25182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:55:12.956051 25182 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:55:12.956990 25182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:55:12.957294 25182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:55:12.958364 25182 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:55:12.960112 25182 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:55:12.960232 25182 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:55:12.960353 25182 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:55:12.960426 25182 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:55:12.997112 25182 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.196:39355
I20240502 01:55:12.997202 25310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.196:39355 every 8 connection(s)
I20240502 01:55:12.998611 25182 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 01:55:13.002146  9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu as pid 25182
I20240502 01:55:13.002382  9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 01:55:13.006734  9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
/tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.197:0
--local_ip_for_outbound_sockets=127.9.176.197
--webserver_interface=127.9.176.197
--webserver_port=0
--tserver_master_addrs=127.9.176.254:38303
--builtin_ntp_servers=127.9.176.212:33577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:55:13.011919 25047 heartbeater.cc:503] Master 127.9.176.254:38303 was elected leader, sending a full tablet report...
I20240502 01:55:13.015269 25311 heartbeater.cc:347] Connected to a master server at 127.9.176.254:38303
I20240502 01:55:13.015764 25311 heartbeater.cc:464] Registering TS with master...
I20240502 01:55:13.017138 25311 heartbeater.cc:511] Master 127.9.176.254:38303 requested a full tablet report, sending...
I20240502 01:55:13.085386 24727 ts_manager.cc:194] Registered new tserver with Master: 30921110704f47f88d68a7d9364d42cc (127.9.176.196:39355)
I20240502 01:55:13.086524 24727 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.176.196:53809
W20240502 01:55:13.387542 25315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:55:13.387894 25315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:55:13.388041 25315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:55:13.396114 25315 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:55:13.396265 25315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:55:13.396453 25315 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.197
I20240502 01:55:13.406564 25315 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:33577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.197:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.9.176.197
--webserver_port=0
--tserver_master_addrs=127.9.176.254:38303
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.197
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:55:13.408290 25315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:55:13.410164 25315 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:55:13.423393 25321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:13.423830 25324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:13.427304 25322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:55:13.429404 25315 server_base.cc:1034] running on GCE node
I20240502 01:55:13.430128 25315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:55:13.431190 25315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:55:13.432411 25315 hybrid_clock.cc:648] HybridClock initialized: now 1714614913432358 us; error 58 us; skew 500 ppm
I20240502 01:55:13.432754 25315 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:55:13.435204 25315 webserver.cc:460] Webserver started at http://127.9.176.197:42303/ using document root <none> and password file <none>
I20240502 01:55:13.436031 25315 fs_manager.cc:359] Metadata directory not provided
I20240502 01:55:13.436168 25315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:55:13.436509 25315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:55:13.439106 25315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "6742b9db50c64ea394c2ca38ddee0872"
format_stamp: "Formatted at 2024-05-02 01:55:13 on dist-test-slave-j3hh"
I20240502 01:55:13.439994 25315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "6742b9db50c64ea394c2ca38ddee0872"
format_stamp: "Formatted at 2024-05-02 01:55:13 on dist-test-slave-j3hh"
I20240502 01:55:13.445636 25315 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.000s
I20240502 01:55:13.449431 25330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:55:13.450966 25315 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.000s
I20240502 01:55:13.451129 25315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "6742b9db50c64ea394c2ca38ddee0872"
format_stamp: "Formatted at 2024-05-02 01:55:13 on dist-test-slave-j3hh"
I20240502 01:55:13.451326 25315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:55:13.475359 25315 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:55:13.476420 25315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:55:13.476702 25315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:55:13.477803 25315 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:55:13.479475 25315 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:55:13.479638 25315 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.001s	sys 0.000s
I20240502 01:55:13.479789 25315 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:55:13.479879 25315 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:55:13.514158 25315 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.197:43479
I20240502 01:55:13.514168 25443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.197:43479 every 8 connection(s)
I20240502 01:55:13.515710 25315 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 01:55:13.521607  9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu as pid 25315
I20240502 01:55:13.521831  9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 01:55:13.525583  9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
/tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.198:0
--local_ip_for_outbound_sockets=127.9.176.198
--webserver_interface=127.9.176.198
--webserver_port=0
--tserver_master_addrs=127.9.176.254:38303
--builtin_ntp_servers=127.9.176.212:33577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:55:13.529878 25444 heartbeater.cc:347] Connected to a master server at 127.9.176.254:38303
I20240502 01:55:13.530259 25444 heartbeater.cc:464] Registering TS with master...
I20240502 01:55:13.531358 25444 heartbeater.cc:511] Master 127.9.176.254:38303 requested a full tablet report, sending...
I20240502 01:55:13.538290 25179 heartbeater.cc:503] Master 127.9.176.254:38303 was elected leader, sending a full tablet report...
I20240502 01:55:13.578195 24727 ts_manager.cc:194] Registered new tserver with Master: 6742b9db50c64ea394c2ca38ddee0872 (127.9.176.197:43479)
I20240502 01:55:13.579279 24727 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.176.197:52693
W20240502 01:55:13.891021 25448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:55:13.891376 25448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:55:13.891554 25448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:55:13.899523 25448 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:55:13.899721 25448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:55:13.899889 25448 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.198
I20240502 01:55:13.909719 25448 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:33577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.198:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.9.176.198
--webserver_port=0
--tserver_master_addrs=127.9.176.254:38303
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.198
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:55:13.911244 25448 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:55:13.913143 25448 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:55:13.926159 25454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:13.926739 25457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:13.927142 25455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:55:13.927870 25448 server_base.cc:1034] running on GCE node
I20240502 01:55:13.928494 25448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:55:13.929522 25448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:55:13.930753 25448 hybrid_clock.cc:648] HybridClock initialized: now 1714614913930700 us; error 46 us; skew 500 ppm
I20240502 01:55:13.931084 25448 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:55:13.933552 25448 webserver.cc:460] Webserver started at http://127.9.176.198:43655/ using document root <none> and password file <none>
I20240502 01:55:13.934337 25448 fs_manager.cc:359] Metadata directory not provided
I20240502 01:55:13.934469 25448 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:55:13.934805 25448 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:55:13.937650 25448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "d490b54f0e954ef59deb6750523f6027"
format_stamp: "Formatted at 2024-05-02 01:55:13 on dist-test-slave-j3hh"
I20240502 01:55:13.938529 25448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "d490b54f0e954ef59deb6750523f6027"
format_stamp: "Formatted at 2024-05-02 01:55:13 on dist-test-slave-j3hh"
I20240502 01:55:13.945531 25448 fs_manager.cc:691] Time spent creating directory manager: real 0.007s	user 0.008s	sys 0.000s
I20240502 01:55:13.949522 25463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:55:13.951010 25448 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.000s
I20240502 01:55:13.951195 25448 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "d490b54f0e954ef59deb6750523f6027"
format_stamp: "Formatted at 2024-05-02 01:55:13 on dist-test-slave-j3hh"
I20240502 01:55:13.951419 25448 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:55:13.977946 25448 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:55:13.978929 25448 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:55:13.979275 25448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:55:13.980435 25448 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:55:13.982081 25448 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:55:13.982194 25448 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:55:13.982323 25448 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:55:13.982406 25448 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:55:14.018220 25448 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.198:34317
I20240502 01:55:14.018298 25576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.198:34317 every 8 connection(s)
I20240502 01:55:14.019701 25448 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 01:55:14.022145  9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu as pid 25448
I20240502 01:55:14.022365  9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240502 01:55:14.033116 25577 heartbeater.cc:347] Connected to a master server at 127.9.176.254:38303
I20240502 01:55:14.033474 25577 heartbeater.cc:464] Registering TS with master...
I20240502 01:55:14.034363 25577 heartbeater.cc:511] Master 127.9.176.254:38303 requested a full tablet report, sending...
I20240502 01:55:14.087002 24727 ts_manager.cc:194] Registered new tserver with Master: d490b54f0e954ef59deb6750523f6027 (127.9.176.198:34317)
I20240502 01:55:14.088078 24727 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.176.198:44589
I20240502 01:55:14.089605 25311 heartbeater.cc:503] Master 127.9.176.254:38303 was elected leader, sending a full tablet report...
I20240502 01:55:14.101604  9923 external_mini_cluster.cc:930] 6 TS(s) registered with all masters
I20240502 01:55:14.124671  9923 external_mini_cluster.cc:930] 6 TS(s) registered with all masters
I20240502 01:55:14.124920  9923 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu pid 25182
I20240502 01:55:14.134048 25306 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:55:14.218690  9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu with pid 25182
I20240502 01:55:14.236496  9923 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu pid 25448
I20240502 01:55:14.244938 25571 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:55:14.329031  9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu with pid 25448
I20240502 01:55:14.346465  9923 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu pid 25315
I20240502 01:55:14.354768 25439 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:55:14.438865  9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu with pid 25315
I20240502 01:55:18.208268 24727 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53224:
name: "rebalance_test_table_0"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
W20240502 01:55:18.215312 24727 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 01:55:18.246474 25112 tablet_service.cc:1467] Processing CreateTablet for tablet ce37ee2e946b4075b78f3c9456939848 (DEFAULT_TABLE table=rebalance_test_table_0 [id=a39bf554fb4649d8983f51465c4cd34d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:18.246068 25114 tablet_service.cc:1467] Processing CreateTablet for tablet f77025fd71924964afafcd32c9ed6923 (DEFAULT_TABLE table=rebalance_test_table_0 [id=a39bf554fb4649d8983f51465c4cd34d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:18.247813 25114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f77025fd71924964afafcd32c9ed6923. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:18.245927 25113 tablet_service.cc:1467] Processing CreateTablet for tablet 1b76f351e8644d9e8ff27e3f1a963ba2 (DEFAULT_TABLE table=rebalance_test_table_0 [id=a39bf554fb4649d8983f51465c4cd34d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:18.250248 25112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce37ee2e946b4075b78f3c9456939848. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:18.249770 24981 tablet_service.cc:1467] Processing CreateTablet for tablet 1b76f351e8644d9e8ff27e3f1a963ba2 (DEFAULT_TABLE table=rebalance_test_table_0 [id=a39bf554fb4649d8983f51465c4cd34d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:18.250370 24980 tablet_service.cc:1467] Processing CreateTablet for tablet ce37ee2e946b4075b78f3c9456939848 (DEFAULT_TABLE table=rebalance_test_table_0 [id=a39bf554fb4649d8983f51465c4cd34d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:18.250975 24982 tablet_service.cc:1467] Processing CreateTablet for tablet f77025fd71924964afafcd32c9ed6923 (DEFAULT_TABLE table=rebalance_test_table_0 [id=a39bf554fb4649d8983f51465c4cd34d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:18.251432 24980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce37ee2e946b4075b78f3c9456939848. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:18.251945 24849 tablet_service.cc:1467] Processing CreateTablet for tablet f77025fd71924964afafcd32c9ed6923 (DEFAULT_TABLE table=rebalance_test_table_0 [id=a39bf554fb4649d8983f51465c4cd34d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:18.252769 24982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f77025fd71924964afafcd32c9ed6923. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:18.253424 24849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f77025fd71924964afafcd32c9ed6923. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:18.253540 25113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b76f351e8644d9e8ff27e3f1a963ba2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:18.254135 24848 tablet_service.cc:1467] Processing CreateTablet for tablet 1b76f351e8644d9e8ff27e3f1a963ba2 (DEFAULT_TABLE table=rebalance_test_table_0 [id=a39bf554fb4649d8983f51465c4cd34d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:18.252099 24847 tablet_service.cc:1467] Processing CreateTablet for tablet ce37ee2e946b4075b78f3c9456939848 (DEFAULT_TABLE table=rebalance_test_table_0 [id=a39bf554fb4649d8983f51465c4cd34d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:18.254865 24981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b76f351e8644d9e8ff27e3f1a963ba2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:18.258710 24848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b76f351e8644d9e8ff27e3f1a963ba2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:18.259680 24847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce37ee2e946b4075b78f3c9456939848. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:18.283154 25605 tablet_bootstrap.cc:492] T f77025fd71924964afafcd32c9ed6923 P 75bd3f0e8d7c4e8aa92a353301afeb21: Bootstrap starting.
I20240502 01:55:18.284658 25603 tablet_bootstrap.cc:492] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 672a55e7d4874c5d99f3c5e3d9550f15: Bootstrap starting.
I20240502 01:55:18.287497 25604 tablet_bootstrap.cc:492] T f77025fd71924964afafcd32c9ed6923 P 96e5c2fd5d624c2da735e72b387a12d9: Bootstrap starting.
I20240502 01:55:18.287667 25605 tablet_bootstrap.cc:654] T f77025fd71924964afafcd32c9ed6923 P 75bd3f0e8d7c4e8aa92a353301afeb21: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:18.289268 25605 log.cc:824] T f77025fd71924964afafcd32c9ed6923 P 75bd3f0e8d7c4e8aa92a353301afeb21: Log is configured to *not* fsync() on all Append() calls
I20240502 01:55:18.290060 25603 tablet_bootstrap.cc:654] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 672a55e7d4874c5d99f3c5e3d9550f15: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:18.291133 25604 tablet_bootstrap.cc:654] T f77025fd71924964afafcd32c9ed6923 P 96e5c2fd5d624c2da735e72b387a12d9: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:18.291445 25603 log.cc:824] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 672a55e7d4874c5d99f3c5e3d9550f15: Log is configured to *not* fsync() on all Append() calls
I20240502 01:55:18.292407 25604 log.cc:824] T f77025fd71924964afafcd32c9ed6923 P 96e5c2fd5d624c2da735e72b387a12d9: Log is configured to *not* fsync() on all Append() calls
I20240502 01:55:18.292619 25605 tablet_bootstrap.cc:492] T f77025fd71924964afafcd32c9ed6923 P 75bd3f0e8d7c4e8aa92a353301afeb21: No bootstrap required, opened a new log
I20240502 01:55:18.292937 25605 ts_tablet_manager.cc:1367] T f77025fd71924964afafcd32c9ed6923 P 75bd3f0e8d7c4e8aa92a353301afeb21: Time spent bootstrapping tablet: real 0.010s	user 0.007s	sys 0.001s
I20240502 01:55:18.294150 25603 tablet_bootstrap.cc:492] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 672a55e7d4874c5d99f3c5e3d9550f15: No bootstrap required, opened a new log
I20240502 01:55:18.294410 25603 ts_tablet_manager.cc:1367] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 672a55e7d4874c5d99f3c5e3d9550f15: Time spent bootstrapping tablet: real 0.010s	user 0.000s	sys 0.006s
I20240502 01:55:18.295403 25604 tablet_bootstrap.cc:492] T f77025fd71924964afafcd32c9ed6923 P 96e5c2fd5d624c2da735e72b387a12d9: No bootstrap required, opened a new log
I20240502 01:55:18.295697 25604 ts_tablet_manager.cc:1367] T f77025fd71924964afafcd32c9ed6923 P 96e5c2fd5d624c2da735e72b387a12d9: Time spent bootstrapping tablet: real 0.008s	user 0.008s	sys 0.000s
I20240502 01:55:18.300769 25603 raft_consensus.cc:348] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } }
I20240502 01:55:18.301290 25603 raft_consensus.cc:374] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:18.301402 25603 raft_consensus.cc:729] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 672a55e7d4874c5d99f3c5e3d9550f15, State: Initialized, Role: FOLLOWER
I20240502 01:55:18.302109 25603 consensus_queue.cc:260] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 672a55e7d4874c5d99f3c5e3d9550f15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } }
I20240502 01:55:18.302160 25605 raft_consensus.cc:348] T f77025fd71924964afafcd32c9ed6923 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } }
I20240502 01:55:18.302608 25605 raft_consensus.cc:374] T f77025fd71924964afafcd32c9ed6923 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:18.302717 25605 raft_consensus.cc:729] T f77025fd71924964afafcd32c9ed6923 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75bd3f0e8d7c4e8aa92a353301afeb21, State: Initialized, Role: FOLLOWER
I20240502 01:55:18.303387 25605 consensus_queue.cc:260] T f77025fd71924964afafcd32c9ed6923 P 75bd3f0e8d7c4e8aa92a353301afeb21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } }
I20240502 01:55:18.305418 25603 ts_tablet_manager.cc:1398] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 672a55e7d4874c5d99f3c5e3d9550f15: Time spent starting tablet: real 0.011s	user 0.012s	sys 0.000s
I20240502 01:55:18.305860 25603 tablet_bootstrap.cc:492] T ce37ee2e946b4075b78f3c9456939848 P 672a55e7d4874c5d99f3c5e3d9550f15: Bootstrap starting.
I20240502 01:55:18.305985 25605 ts_tablet_manager.cc:1398] T f77025fd71924964afafcd32c9ed6923 P 75bd3f0e8d7c4e8aa92a353301afeb21: Time spent starting tablet: real 0.013s	user 0.011s	sys 0.003s
I20240502 01:55:18.306406 25605 tablet_bootstrap.cc:492] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 75bd3f0e8d7c4e8aa92a353301afeb21: Bootstrap starting.
I20240502 01:55:18.304890 25604 raft_consensus.cc:348] T f77025fd71924964afafcd32c9ed6923 P 96e5c2fd5d624c2da735e72b387a12d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } }
I20240502 01:55:18.306671 25604 raft_consensus.cc:374] T f77025fd71924964afafcd32c9ed6923 P 96e5c2fd5d624c2da735e72b387a12d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:18.306768 25604 raft_consensus.cc:729] T f77025fd71924964afafcd32c9ed6923 P 96e5c2fd5d624c2da735e72b387a12d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96e5c2fd5d624c2da735e72b387a12d9, State: Initialized, Role: FOLLOWER
I20240502 01:55:18.307453 25604 consensus_queue.cc:260] T f77025fd71924964afafcd32c9ed6923 P 96e5c2fd5d624c2da735e72b387a12d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } }
I20240502 01:55:18.308154 25603 tablet_bootstrap.cc:654] T ce37ee2e946b4075b78f3c9456939848 P 672a55e7d4874c5d99f3c5e3d9550f15: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:18.310160 25605 tablet_bootstrap.cc:654] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 75bd3f0e8d7c4e8aa92a353301afeb21: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:18.314914 25603 tablet_bootstrap.cc:492] T ce37ee2e946b4075b78f3c9456939848 P 672a55e7d4874c5d99f3c5e3d9550f15: No bootstrap required, opened a new log
I20240502 01:55:18.315088 25603 ts_tablet_manager.cc:1367] T ce37ee2e946b4075b78f3c9456939848 P 672a55e7d4874c5d99f3c5e3d9550f15: Time spent bootstrapping tablet: real 0.009s	user 0.003s	sys 0.004s
I20240502 01:55:18.315491 25604 ts_tablet_manager.cc:1398] T f77025fd71924964afafcd32c9ed6923 P 96e5c2fd5d624c2da735e72b387a12d9: Time spent starting tablet: real 0.020s	user 0.009s	sys 0.007s
I20240502 01:55:18.315997 25604 tablet_bootstrap.cc:492] T ce37ee2e946b4075b78f3c9456939848 P 96e5c2fd5d624c2da735e72b387a12d9: Bootstrap starting.
I20240502 01:55:18.316190 25605 tablet_bootstrap.cc:492] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 75bd3f0e8d7c4e8aa92a353301afeb21: No bootstrap required, opened a new log
I20240502 01:55:18.316426 25605 ts_tablet_manager.cc:1367] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 75bd3f0e8d7c4e8aa92a353301afeb21: Time spent bootstrapping tablet: real 0.010s	user 0.006s	sys 0.001s
I20240502 01:55:18.316287 25603 raft_consensus.cc:348] T ce37ee2e946b4075b78f3c9456939848 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } }
I20240502 01:55:18.316516 25603 raft_consensus.cc:374] T ce37ee2e946b4075b78f3c9456939848 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:18.316615 25603 raft_consensus.cc:729] T ce37ee2e946b4075b78f3c9456939848 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 672a55e7d4874c5d99f3c5e3d9550f15, State: Initialized, Role: FOLLOWER
I20240502 01:55:18.316916 25603 consensus_queue.cc:260] T ce37ee2e946b4075b78f3c9456939848 P 672a55e7d4874c5d99f3c5e3d9550f15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } }
I20240502 01:55:18.317617 25603 ts_tablet_manager.cc:1398] T ce37ee2e946b4075b78f3c9456939848 P 672a55e7d4874c5d99f3c5e3d9550f15: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 01:55:18.317796 25604 tablet_bootstrap.cc:654] T ce37ee2e946b4075b78f3c9456939848 P 96e5c2fd5d624c2da735e72b387a12d9: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:18.317922 25603 tablet_bootstrap.cc:492] T f77025fd71924964afafcd32c9ed6923 P 672a55e7d4874c5d99f3c5e3d9550f15: Bootstrap starting.
I20240502 01:55:18.318737 25605 raft_consensus.cc:348] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } }
I20240502 01:55:18.319067 25605 raft_consensus.cc:374] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:18.319208 25605 raft_consensus.cc:729] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75bd3f0e8d7c4e8aa92a353301afeb21, State: Initialized, Role: FOLLOWER
I20240502 01:55:18.319677 25603 tablet_bootstrap.cc:654] T f77025fd71924964afafcd32c9ed6923 P 672a55e7d4874c5d99f3c5e3d9550f15: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:18.319609 25605 consensus_queue.cc:260] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 75bd3f0e8d7c4e8aa92a353301afeb21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } }
I20240502 01:55:18.320325 25605 ts_tablet_manager.cc:1398] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 75bd3f0e8d7c4e8aa92a353301afeb21: Time spent starting tablet: real 0.004s	user 0.000s	sys 0.002s
I20240502 01:55:18.320657 25605 tablet_bootstrap.cc:492] T ce37ee2e946b4075b78f3c9456939848 P 75bd3f0e8d7c4e8aa92a353301afeb21: Bootstrap starting.
I20240502 01:55:18.323057 25605 tablet_bootstrap.cc:654] T ce37ee2e946b4075b78f3c9456939848 P 75bd3f0e8d7c4e8aa92a353301afeb21: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:18.328616 25603 tablet_bootstrap.cc:492] T f77025fd71924964afafcd32c9ed6923 P 672a55e7d4874c5d99f3c5e3d9550f15: No bootstrap required, opened a new log
I20240502 01:55:18.328888 25603 ts_tablet_manager.cc:1367] T f77025fd71924964afafcd32c9ed6923 P 672a55e7d4874c5d99f3c5e3d9550f15: Time spent bootstrapping tablet: real 0.011s	user 0.001s	sys 0.008s
I20240502 01:55:18.330027 25603 raft_consensus.cc:348] T f77025fd71924964afafcd32c9ed6923 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } }
I20240502 01:55:18.330415 25603 raft_consensus.cc:374] T f77025fd71924964afafcd32c9ed6923 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:18.330503 25603 raft_consensus.cc:729] T f77025fd71924964afafcd32c9ed6923 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 672a55e7d4874c5d99f3c5e3d9550f15, State: Initialized, Role: FOLLOWER
I20240502 01:55:18.330957 25603 consensus_queue.cc:260] T f77025fd71924964afafcd32c9ed6923 P 672a55e7d4874c5d99f3c5e3d9550f15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } }
I20240502 01:55:18.331929 25603 ts_tablet_manager.cc:1398] T f77025fd71924964afafcd32c9ed6923 P 672a55e7d4874c5d99f3c5e3d9550f15: Time spent starting tablet: real 0.003s	user 0.004s	sys 0.000s
I20240502 01:55:18.333182 25605 tablet_bootstrap.cc:492] T ce37ee2e946b4075b78f3c9456939848 P 75bd3f0e8d7c4e8aa92a353301afeb21: No bootstrap required, opened a new log
I20240502 01:55:18.333388 25605 ts_tablet_manager.cc:1367] T ce37ee2e946b4075b78f3c9456939848 P 75bd3f0e8d7c4e8aa92a353301afeb21: Time spent bootstrapping tablet: real 0.013s	user 0.005s	sys 0.007s
I20240502 01:55:18.334151 25604 tablet_bootstrap.cc:492] T ce37ee2e946b4075b78f3c9456939848 P 96e5c2fd5d624c2da735e72b387a12d9: No bootstrap required, opened a new log
I20240502 01:55:18.334200 25610 raft_consensus.cc:482] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:55:18.334352 25604 ts_tablet_manager.cc:1367] T ce37ee2e946b4075b78f3c9456939848 P 96e5c2fd5d624c2da735e72b387a12d9: Time spent bootstrapping tablet: real 0.018s	user 0.006s	sys 0.011s
I20240502 01:55:18.334458 25610 raft_consensus.cc:504] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } }
I20240502 01:55:18.335536 25604 raft_consensus.cc:348] T ce37ee2e946b4075b78f3c9456939848 P 96e5c2fd5d624c2da735e72b387a12d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } }
I20240502 01:55:18.335801 25604 raft_consensus.cc:374] T ce37ee2e946b4075b78f3c9456939848 P 96e5c2fd5d624c2da735e72b387a12d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:18.335906 25604 raft_consensus.cc:729] T ce37ee2e946b4075b78f3c9456939848 P 96e5c2fd5d624c2da735e72b387a12d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96e5c2fd5d624c2da735e72b387a12d9, State: Initialized, Role: FOLLOWER
I20240502 01:55:18.336190 25605 raft_consensus.cc:348] T ce37ee2e946b4075b78f3c9456939848 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } }
I20240502 01:55:18.336216 25604 consensus_queue.cc:260] T ce37ee2e946b4075b78f3c9456939848 P 96e5c2fd5d624c2da735e72b387a12d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } }
I20240502 01:55:18.336416 25605 raft_consensus.cc:374] T ce37ee2e946b4075b78f3c9456939848 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:18.336504 25605 raft_consensus.cc:729] T ce37ee2e946b4075b78f3c9456939848 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75bd3f0e8d7c4e8aa92a353301afeb21, State: Initialized, Role: FOLLOWER
I20240502 01:55:18.336809 25604 ts_tablet_manager.cc:1398] T ce37ee2e946b4075b78f3c9456939848 P 96e5c2fd5d624c2da735e72b387a12d9: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 01:55:18.336882 25610 leader_election.cc:288] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 75bd3f0e8d7c4e8aa92a353301afeb21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96e5c2fd5d624c2da735e72b387a12d9 (127.9.176.193:32783), 672a55e7d4874c5d99f3c5e3d9550f15 (127.9.176.195:46579)
I20240502 01:55:18.336781 25605 consensus_queue.cc:260] T ce37ee2e946b4075b78f3c9456939848 P 75bd3f0e8d7c4e8aa92a353301afeb21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } }
I20240502 01:55:18.337162 25604 tablet_bootstrap.cc:492] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 96e5c2fd5d624c2da735e72b387a12d9: Bootstrap starting.
I20240502 01:55:18.337546 25605 ts_tablet_manager.cc:1398] T ce37ee2e946b4075b78f3c9456939848 P 75bd3f0e8d7c4e8aa92a353301afeb21: Time spent starting tablet: real 0.004s	user 0.001s	sys 0.000s
I20240502 01:55:18.338950 25604 tablet_bootstrap.cc:654] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 96e5c2fd5d624c2da735e72b387a12d9: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:18.344084 25604 tablet_bootstrap.cc:492] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 96e5c2fd5d624c2da735e72b387a12d9: No bootstrap required, opened a new log
I20240502 01:55:18.344323 25604 ts_tablet_manager.cc:1367] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 96e5c2fd5d624c2da735e72b387a12d9: Time spent bootstrapping tablet: real 0.007s	user 0.002s	sys 0.004s
I20240502 01:55:18.345194 25604 raft_consensus.cc:348] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 96e5c2fd5d624c2da735e72b387a12d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } }
I20240502 01:55:18.345448 25604 raft_consensus.cc:374] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 96e5c2fd5d624c2da735e72b387a12d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:18.345530 25604 raft_consensus.cc:729] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 96e5c2fd5d624c2da735e72b387a12d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96e5c2fd5d624c2da735e72b387a12d9, State: Initialized, Role: FOLLOWER
I20240502 01:55:18.345818 25604 consensus_queue.cc:260] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 96e5c2fd5d624c2da735e72b387a12d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } }
I20240502 01:55:18.346484 25604 ts_tablet_manager.cc:1398] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 96e5c2fd5d624c2da735e72b387a12d9: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 01:55:18.349714 24869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b76f351e8644d9e8ff27e3f1a963ba2" candidate_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96e5c2fd5d624c2da735e72b387a12d9" is_pre_election: true
I20240502 01:55:18.350414 24869 raft_consensus.cc:2454] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 96e5c2fd5d624c2da735e72b387a12d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75bd3f0e8d7c4e8aa92a353301afeb21 in term 0.
I20240502 01:55:18.351240 24935 leader_election.cc:302] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 75bd3f0e8d7c4e8aa92a353301afeb21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75bd3f0e8d7c4e8aa92a353301afeb21, 96e5c2fd5d624c2da735e72b387a12d9; no voters: 
I20240502 01:55:18.351711 25610 raft_consensus.cc:2781] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:55:18.351917 25610 raft_consensus.cc:482] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:55:18.352042 25610 raft_consensus.cc:3037] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:18.352535 25134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b76f351e8644d9e8ff27e3f1a963ba2" candidate_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" is_pre_election: true
I20240502 01:55:18.353070 25134 raft_consensus.cc:2454] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75bd3f0e8d7c4e8aa92a353301afeb21 in term 0.
I20240502 01:55:18.354511 25610 raft_consensus.cc:504] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } }
I20240502 01:55:18.355100 25610 leader_election.cc:288] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 75bd3f0e8d7c4e8aa92a353301afeb21 [CANDIDATE]: Term 1 election: Requested vote from peers 96e5c2fd5d624c2da735e72b387a12d9 (127.9.176.193:32783), 672a55e7d4874c5d99f3c5e3d9550f15 (127.9.176.195:46579)
I20240502 01:55:18.355646 24869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b76f351e8644d9e8ff27e3f1a963ba2" candidate_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96e5c2fd5d624c2da735e72b387a12d9"
I20240502 01:55:18.355763 25134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b76f351e8644d9e8ff27e3f1a963ba2" candidate_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "672a55e7d4874c5d99f3c5e3d9550f15"
I20240502 01:55:18.355899 24869 raft_consensus.cc:3037] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 96e5c2fd5d624c2da735e72b387a12d9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:18.355965 25134 raft_consensus.cc:3037] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:18.359161 24869 raft_consensus.cc:2454] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 96e5c2fd5d624c2da735e72b387a12d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75bd3f0e8d7c4e8aa92a353301afeb21 in term 1.
I20240502 01:55:18.359161 25134 raft_consensus.cc:2454] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75bd3f0e8d7c4e8aa92a353301afeb21 in term 1.
I20240502 01:55:18.359709 24935 leader_election.cc:302] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 75bd3f0e8d7c4e8aa92a353301afeb21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75bd3f0e8d7c4e8aa92a353301afeb21, 96e5c2fd5d624c2da735e72b387a12d9; no voters: 
I20240502 01:55:18.360044 25610 raft_consensus.cc:2781] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:55:18.360497 25610 raft_consensus.cc:686] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 1 LEADER]: Becoming Leader. State: Replica: 75bd3f0e8d7c4e8aa92a353301afeb21, State: Running, Role: LEADER
I20240502 01:55:18.360970 25610 consensus_queue.cc:237] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 75bd3f0e8d7c4e8aa92a353301afeb21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } }
I20240502 01:55:18.366115 24727 catalog_manager.cc:5484] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 75bd3f0e8d7c4e8aa92a353301afeb21 reported cstate change: term changed from 0 to 1, leader changed from <none> to 75bd3f0e8d7c4e8aa92a353301afeb21 (127.9.176.194). New cstate: current_term: 1 leader_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } health_report { overall_health: UNKNOWN } } }
I20240502 01:55:18.426720 25610 raft_consensus.cc:482] T f77025fd71924964afafcd32c9ed6923 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:55:18.426956 25610 raft_consensus.cc:504] T f77025fd71924964afafcd32c9ed6923 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } }
I20240502 01:55:18.427553 25610 leader_election.cc:288] T f77025fd71924964afafcd32c9ed6923 P 75bd3f0e8d7c4e8aa92a353301afeb21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 672a55e7d4874c5d99f3c5e3d9550f15 (127.9.176.195:46579), 96e5c2fd5d624c2da735e72b387a12d9 (127.9.176.193:32783)
I20240502 01:55:18.428264 25134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f77025fd71924964afafcd32c9ed6923" candidate_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" is_pre_election: true
I20240502 01:55:18.428300 24869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f77025fd71924964afafcd32c9ed6923" candidate_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96e5c2fd5d624c2da735e72b387a12d9" is_pre_election: true
I20240502 01:55:18.428573 25134 raft_consensus.cc:2454] T f77025fd71924964afafcd32c9ed6923 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75bd3f0e8d7c4e8aa92a353301afeb21 in term 0.
I20240502 01:55:18.428573 24869 raft_consensus.cc:2454] T f77025fd71924964afafcd32c9ed6923 P 96e5c2fd5d624c2da735e72b387a12d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75bd3f0e8d7c4e8aa92a353301afeb21 in term 0.
I20240502 01:55:18.429010 24935 leader_election.cc:302] T f77025fd71924964afafcd32c9ed6923 P 75bd3f0e8d7c4e8aa92a353301afeb21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75bd3f0e8d7c4e8aa92a353301afeb21, 96e5c2fd5d624c2da735e72b387a12d9; no voters: 
I20240502 01:55:18.429315 25610 raft_consensus.cc:2781] T f77025fd71924964afafcd32c9ed6923 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:55:18.429425 25610 raft_consensus.cc:482] T f77025fd71924964afafcd32c9ed6923 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:55:18.429535 25610 raft_consensus.cc:3037] T f77025fd71924964afafcd32c9ed6923 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:18.431495 25610 raft_consensus.cc:504] T f77025fd71924964afafcd32c9ed6923 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } }
I20240502 01:55:18.432264 25610 leader_election.cc:288] T f77025fd71924964afafcd32c9ed6923 P 75bd3f0e8d7c4e8aa92a353301afeb21 [CANDIDATE]: Term 1 election: Requested vote from peers 672a55e7d4874c5d99f3c5e3d9550f15 (127.9.176.195:46579), 96e5c2fd5d624c2da735e72b387a12d9 (127.9.176.193:32783)
I20240502 01:55:18.432662 25134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f77025fd71924964afafcd32c9ed6923" candidate_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "672a55e7d4874c5d99f3c5e3d9550f15"
I20240502 01:55:18.432722 24869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f77025fd71924964afafcd32c9ed6923" candidate_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96e5c2fd5d624c2da735e72b387a12d9"
I20240502 01:55:18.432888 25134 raft_consensus.cc:3037] T f77025fd71924964afafcd32c9ed6923 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:18.432956 24869 raft_consensus.cc:3037] T f77025fd71924964afafcd32c9ed6923 P 96e5c2fd5d624c2da735e72b387a12d9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:18.434804 25134 raft_consensus.cc:2454] T f77025fd71924964afafcd32c9ed6923 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75bd3f0e8d7c4e8aa92a353301afeb21 in term 1.
I20240502 01:55:18.434867 24869 raft_consensus.cc:2454] T f77025fd71924964afafcd32c9ed6923 P 96e5c2fd5d624c2da735e72b387a12d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75bd3f0e8d7c4e8aa92a353301afeb21 in term 1.
I20240502 01:55:18.435228 24935 leader_election.cc:302] T f77025fd71924964afafcd32c9ed6923 P 75bd3f0e8d7c4e8aa92a353301afeb21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 672a55e7d4874c5d99f3c5e3d9550f15, 75bd3f0e8d7c4e8aa92a353301afeb21; no voters: 
I20240502 01:55:18.435519 25610 raft_consensus.cc:2781] T f77025fd71924964afafcd32c9ed6923 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:55:18.435729 25610 raft_consensus.cc:686] T f77025fd71924964afafcd32c9ed6923 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 1 LEADER]: Becoming Leader. State: Replica: 75bd3f0e8d7c4e8aa92a353301afeb21, State: Running, Role: LEADER
I20240502 01:55:18.436018 25610 consensus_queue.cc:237] T f77025fd71924964afafcd32c9ed6923 P 75bd3f0e8d7c4e8aa92a353301afeb21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } }
I20240502 01:55:18.438474 24727 catalog_manager.cc:5484] T f77025fd71924964afafcd32c9ed6923 P 75bd3f0e8d7c4e8aa92a353301afeb21 reported cstate change: term changed from 0 to 1, leader changed from <none> to 75bd3f0e8d7c4e8aa92a353301afeb21 (127.9.176.194). New cstate: current_term: 1 leader_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } health_report { overall_health: UNKNOWN } } }
W20240502 01:55:18.449415 25048 tablet.cc:2369] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 75bd3f0e8d7c4e8aa92a353301afeb21: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 01:55:18.452245 24915 tablet.cc:2369] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 96e5c2fd5d624c2da735e72b387a12d9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 01:55:18.456576 25180 tablet.cc:2369] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 672a55e7d4874c5d99f3c5e3d9550f15: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:55:18.639147 25609 raft_consensus.cc:482] T ce37ee2e946b4075b78f3c9456939848 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:55:18.639379 25609 raft_consensus.cc:504] T ce37ee2e946b4075b78f3c9456939848 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } }
I20240502 01:55:18.641093 25609 leader_election.cc:288] T ce37ee2e946b4075b78f3c9456939848 P 672a55e7d4874c5d99f3c5e3d9550f15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96e5c2fd5d624c2da735e72b387a12d9 (127.9.176.193:32783), 75bd3f0e8d7c4e8aa92a353301afeb21 (127.9.176.194:41625)
I20240502 01:55:18.648468 24869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce37ee2e946b4075b78f3c9456939848" candidate_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96e5c2fd5d624c2da735e72b387a12d9" is_pre_election: true
I20240502 01:55:18.648716 24869 raft_consensus.cc:2454] T ce37ee2e946b4075b78f3c9456939848 P 96e5c2fd5d624c2da735e72b387a12d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 672a55e7d4874c5d99f3c5e3d9550f15 in term 0.
I20240502 01:55:18.648682 25002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce37ee2e946b4075b78f3c9456939848" candidate_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" is_pre_election: true
I20240502 01:55:18.649142 25002 raft_consensus.cc:2454] T ce37ee2e946b4075b78f3c9456939848 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 672a55e7d4874c5d99f3c5e3d9550f15 in term 0.
I20240502 01:55:18.649263 25068 leader_election.cc:302] T ce37ee2e946b4075b78f3c9456939848 P 672a55e7d4874c5d99f3c5e3d9550f15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 672a55e7d4874c5d99f3c5e3d9550f15, 96e5c2fd5d624c2da735e72b387a12d9; no voters: 
I20240502 01:55:18.649706 25609 raft_consensus.cc:2781] T ce37ee2e946b4075b78f3c9456939848 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:55:18.649849 25609 raft_consensus.cc:482] T ce37ee2e946b4075b78f3c9456939848 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:55:18.649947 25609 raft_consensus.cc:3037] T ce37ee2e946b4075b78f3c9456939848 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:18.652076 25609 raft_consensus.cc:504] T ce37ee2e946b4075b78f3c9456939848 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } }
I20240502 01:55:18.652698 25609 leader_election.cc:288] T ce37ee2e946b4075b78f3c9456939848 P 672a55e7d4874c5d99f3c5e3d9550f15 [CANDIDATE]: Term 1 election: Requested vote from peers 96e5c2fd5d624c2da735e72b387a12d9 (127.9.176.193:32783), 75bd3f0e8d7c4e8aa92a353301afeb21 (127.9.176.194:41625)
I20240502 01:55:18.653334 24869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce37ee2e946b4075b78f3c9456939848" candidate_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96e5c2fd5d624c2da735e72b387a12d9"
I20240502 01:55:18.653333 25002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce37ee2e946b4075b78f3c9456939848" candidate_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21"
I20240502 01:55:18.653577 24869 raft_consensus.cc:3037] T ce37ee2e946b4075b78f3c9456939848 P 96e5c2fd5d624c2da735e72b387a12d9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:18.653577 25002 raft_consensus.cc:3037] T ce37ee2e946b4075b78f3c9456939848 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:18.656713 25002 raft_consensus.cc:2454] T ce37ee2e946b4075b78f3c9456939848 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 672a55e7d4874c5d99f3c5e3d9550f15 in term 1.
I20240502 01:55:18.656718 24869 raft_consensus.cc:2454] T ce37ee2e946b4075b78f3c9456939848 P 96e5c2fd5d624c2da735e72b387a12d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 672a55e7d4874c5d99f3c5e3d9550f15 in term 1.
I20240502 01:55:18.657306 25068 leader_election.cc:302] T ce37ee2e946b4075b78f3c9456939848 P 672a55e7d4874c5d99f3c5e3d9550f15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 672a55e7d4874c5d99f3c5e3d9550f15, 75bd3f0e8d7c4e8aa92a353301afeb21; no voters: 
I20240502 01:55:18.657650 25609 raft_consensus.cc:2781] T ce37ee2e946b4075b78f3c9456939848 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:55:18.658120 25609 raft_consensus.cc:686] T ce37ee2e946b4075b78f3c9456939848 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 1 LEADER]: Becoming Leader. State: Replica: 672a55e7d4874c5d99f3c5e3d9550f15, State: Running, Role: LEADER
I20240502 01:55:18.658666 25609 consensus_queue.cc:237] T ce37ee2e946b4075b78f3c9456939848 P 672a55e7d4874c5d99f3c5e3d9550f15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } }
I20240502 01:55:18.663599 24727 catalog_manager.cc:5484] T ce37ee2e946b4075b78f3c9456939848 P 672a55e7d4874c5d99f3c5e3d9550f15 reported cstate change: term changed from 0 to 1, leader changed from <none> to 672a55e7d4874c5d99f3c5e3d9550f15 (127.9.176.195). New cstate: current_term: 1 leader_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } health_report { overall_health: HEALTHY } } }
I20240502 01:55:18.787111 25610 consensus_queue.cc:1035] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 75bd3f0e8d7c4e8aa92a353301afeb21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:18.799477 25610 consensus_queue.cc:1035] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 75bd3f0e8d7c4e8aa92a353301afeb21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:19.032438 25610 consensus_queue.cc:1035] T f77025fd71924964afafcd32c9ed6923 P 75bd3f0e8d7c4e8aa92a353301afeb21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:19.038775 25620 consensus_queue.cc:1035] T f77025fd71924964afafcd32c9ed6923 P 75bd3f0e8d7c4e8aa92a353301afeb21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:19.041872 25609 consensus_queue.cc:1035] T ce37ee2e946b4075b78f3c9456939848 P 672a55e7d4874c5d99f3c5e3d9550f15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:19.051657 25625 consensus_queue.cc:1035] T ce37ee2e946b4075b78f3c9456939848 P 672a55e7d4874c5d99f3c5e3d9550f15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:55:19.122133 25627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:55:19.122478 25627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:55:19.130209 25627 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'

INSERT report
    rows total: 2000
    time total: 220.274 ms
  time per row: 0.110137 ms
I20240502 01:55:19.526302 24723 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53224:
name: "rebalance_test_table_1"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
W20240502 01:55:19.527092 24723 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 01:55:19.535667 24849 tablet_service.cc:1467] Processing CreateTablet for tablet 75d71cdcf4d44353acb7d2f18696977b (DEFAULT_TABLE table=rebalance_test_table_1 [id=4e02aab5efa847ab8735d911dc7a4f5a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:19.535887 24980 tablet_service.cc:1467] Processing CreateTablet for tablet 75d71cdcf4d44353acb7d2f18696977b (DEFAULT_TABLE table=rebalance_test_table_1 [id=4e02aab5efa847ab8735d911dc7a4f5a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:19.536218 24849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75d71cdcf4d44353acb7d2f18696977b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:19.536288 24980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75d71cdcf4d44353acb7d2f18696977b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:19.537005 25114 tablet_service.cc:1467] Processing CreateTablet for tablet 450538b8217d40ce945d51e2c171ee36 (DEFAULT_TABLE table=rebalance_test_table_1 [id=4e02aab5efa847ab8735d911dc7a4f5a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:19.537598 25114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 450538b8217d40ce945d51e2c171ee36. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:19.537730 24982 tablet_service.cc:1467] Processing CreateTablet for tablet 450538b8217d40ce945d51e2c171ee36 (DEFAULT_TABLE table=rebalance_test_table_1 [id=4e02aab5efa847ab8735d911dc7a4f5a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:19.538174 24982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 450538b8217d40ce945d51e2c171ee36. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:19.545025 24848 tablet_service.cc:1467] Processing CreateTablet for tablet 450538b8217d40ce945d51e2c171ee36 (DEFAULT_TABLE table=rebalance_test_table_1 [id=4e02aab5efa847ab8735d911dc7a4f5a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:19.545684 24848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 450538b8217d40ce945d51e2c171ee36. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:19.547878 25113 tablet_service.cc:1467] Processing CreateTablet for tablet 75d71cdcf4d44353acb7d2f18696977b (DEFAULT_TABLE table=rebalance_test_table_1 [id=4e02aab5efa847ab8735d911dc7a4f5a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:19.548571 25113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75d71cdcf4d44353acb7d2f18696977b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:19.551285 25689 tablet_bootstrap.cc:492] T 75d71cdcf4d44353acb7d2f18696977b P 75bd3f0e8d7c4e8aa92a353301afeb21: Bootstrap starting.
I20240502 01:55:19.552423 25690 tablet_bootstrap.cc:492] T 450538b8217d40ce945d51e2c171ee36 P 672a55e7d4874c5d99f3c5e3d9550f15: Bootstrap starting.
I20240502 01:55:19.553768 25112 tablet_service.cc:1467] Processing CreateTablet for tablet b6beb3b47fd545d2b6a2d9181f4e5468 (DEFAULT_TABLE table=rebalance_test_table_1 [id=4e02aab5efa847ab8735d911dc7a4f5a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:19.554104 24847 tablet_service.cc:1467] Processing CreateTablet for tablet b6beb3b47fd545d2b6a2d9181f4e5468 (DEFAULT_TABLE table=rebalance_test_table_1 [id=4e02aab5efa847ab8735d911dc7a4f5a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:19.554324 25112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6beb3b47fd545d2b6a2d9181f4e5468. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:19.554574 24847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6beb3b47fd545d2b6a2d9181f4e5468. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:19.554858 25690 tablet_bootstrap.cc:654] T 450538b8217d40ce945d51e2c171ee36 P 672a55e7d4874c5d99f3c5e3d9550f15: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:19.562077 25689 tablet_bootstrap.cc:654] T 75d71cdcf4d44353acb7d2f18696977b P 75bd3f0e8d7c4e8aa92a353301afeb21: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:19.571343 25689 tablet_bootstrap.cc:492] T 75d71cdcf4d44353acb7d2f18696977b P 75bd3f0e8d7c4e8aa92a353301afeb21: No bootstrap required, opened a new log
I20240502 01:55:19.571743 25689 ts_tablet_manager.cc:1367] T 75d71cdcf4d44353acb7d2f18696977b P 75bd3f0e8d7c4e8aa92a353301afeb21: Time spent bootstrapping tablet: real 0.021s	user 0.001s	sys 0.010s
I20240502 01:55:19.572428 24980 tablet_service.cc:1467] Processing CreateTablet for tablet b6beb3b47fd545d2b6a2d9181f4e5468 (DEFAULT_TABLE table=rebalance_test_table_1 [id=4e02aab5efa847ab8735d911dc7a4f5a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:19.573894 24980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6beb3b47fd545d2b6a2d9181f4e5468. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:19.573096 25689 raft_consensus.cc:348] T 75d71cdcf4d44353acb7d2f18696977b P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } }
I20240502 01:55:19.574797 25689 raft_consensus.cc:374] T 75d71cdcf4d44353acb7d2f18696977b P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:19.574976 25689 raft_consensus.cc:729] T 75d71cdcf4d44353acb7d2f18696977b P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75bd3f0e8d7c4e8aa92a353301afeb21, State: Initialized, Role: FOLLOWER
I20240502 01:55:19.575423 25689 consensus_queue.cc:260] T 75d71cdcf4d44353acb7d2f18696977b P 75bd3f0e8d7c4e8aa92a353301afeb21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } }
I20240502 01:55:19.576531 25690 tablet_bootstrap.cc:492] T 450538b8217d40ce945d51e2c171ee36 P 672a55e7d4874c5d99f3c5e3d9550f15: No bootstrap required, opened a new log
I20240502 01:55:19.576719 25690 ts_tablet_manager.cc:1367] T 450538b8217d40ce945d51e2c171ee36 P 672a55e7d4874c5d99f3c5e3d9550f15: Time spent bootstrapping tablet: real 0.024s	user 0.010s	sys 0.010s
I20240502 01:55:19.578352 25689 ts_tablet_manager.cc:1398] T 75d71cdcf4d44353acb7d2f18696977b P 75bd3f0e8d7c4e8aa92a353301afeb21: Time spent starting tablet: real 0.005s	user 0.001s	sys 0.001s
I20240502 01:55:19.578388 25690 raft_consensus.cc:348] T 450538b8217d40ce945d51e2c171ee36 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } }
I20240502 01:55:19.578613 25690 raft_consensus.cc:374] T 450538b8217d40ce945d51e2c171ee36 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:19.578712 25690 raft_consensus.cc:729] T 450538b8217d40ce945d51e2c171ee36 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 672a55e7d4874c5d99f3c5e3d9550f15, State: Initialized, Role: FOLLOWER
I20240502 01:55:19.578964 25690 consensus_queue.cc:260] T 450538b8217d40ce945d51e2c171ee36 P 672a55e7d4874c5d99f3c5e3d9550f15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } }
I20240502 01:55:19.579587 25690 ts_tablet_manager.cc:1398] T 450538b8217d40ce945d51e2c171ee36 P 672a55e7d4874c5d99f3c5e3d9550f15: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.001s
I20240502 01:55:19.580228 25690 tablet_bootstrap.cc:492] T b6beb3b47fd545d2b6a2d9181f4e5468 P 672a55e7d4874c5d99f3c5e3d9550f15: Bootstrap starting.
I20240502 01:55:19.580535 25688 tablet_bootstrap.cc:492] T 75d71cdcf4d44353acb7d2f18696977b P 96e5c2fd5d624c2da735e72b387a12d9: Bootstrap starting.
I20240502 01:55:19.582051 25690 tablet_bootstrap.cc:654] T b6beb3b47fd545d2b6a2d9181f4e5468 P 672a55e7d4874c5d99f3c5e3d9550f15: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:19.582525 25689 tablet_bootstrap.cc:492] T 450538b8217d40ce945d51e2c171ee36 P 75bd3f0e8d7c4e8aa92a353301afeb21: Bootstrap starting.
I20240502 01:55:19.583060 25688 tablet_bootstrap.cc:654] T 75d71cdcf4d44353acb7d2f18696977b P 96e5c2fd5d624c2da735e72b387a12d9: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:19.584725 25689 tablet_bootstrap.cc:654] T 450538b8217d40ce945d51e2c171ee36 P 75bd3f0e8d7c4e8aa92a353301afeb21: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:19.585791 25690 tablet_bootstrap.cc:492] T b6beb3b47fd545d2b6a2d9181f4e5468 P 672a55e7d4874c5d99f3c5e3d9550f15: No bootstrap required, opened a new log
I20240502 01:55:19.586053 25690 ts_tablet_manager.cc:1367] T b6beb3b47fd545d2b6a2d9181f4e5468 P 672a55e7d4874c5d99f3c5e3d9550f15: Time spent bootstrapping tablet: real 0.006s	user 0.003s	sys 0.002s
I20240502 01:55:19.587105 25690 raft_consensus.cc:348] T b6beb3b47fd545d2b6a2d9181f4e5468 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } }
I20240502 01:55:19.587437 25690 raft_consensus.cc:374] T b6beb3b47fd545d2b6a2d9181f4e5468 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:19.587525 25690 raft_consensus.cc:729] T b6beb3b47fd545d2b6a2d9181f4e5468 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 672a55e7d4874c5d99f3c5e3d9550f15, State: Initialized, Role: FOLLOWER
I20240502 01:55:19.587821 25690 consensus_queue.cc:260] T b6beb3b47fd545d2b6a2d9181f4e5468 P 672a55e7d4874c5d99f3c5e3d9550f15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } }
I20240502 01:55:19.588034 25689 tablet_bootstrap.cc:492] T 450538b8217d40ce945d51e2c171ee36 P 75bd3f0e8d7c4e8aa92a353301afeb21: No bootstrap required, opened a new log
I20240502 01:55:19.588063 25688 tablet_bootstrap.cc:492] T 75d71cdcf4d44353acb7d2f18696977b P 96e5c2fd5d624c2da735e72b387a12d9: No bootstrap required, opened a new log
I20240502 01:55:19.588217 25688 ts_tablet_manager.cc:1367] T 75d71cdcf4d44353acb7d2f18696977b P 96e5c2fd5d624c2da735e72b387a12d9: Time spent bootstrapping tablet: real 0.008s	user 0.006s	sys 0.000s
I20240502 01:55:19.588249 25689 ts_tablet_manager.cc:1367] T 450538b8217d40ce945d51e2c171ee36 P 75bd3f0e8d7c4e8aa92a353301afeb21: Time spent bootstrapping tablet: real 0.006s	user 0.001s	sys 0.004s
I20240502 01:55:19.588390 25690 ts_tablet_manager.cc:1398] T b6beb3b47fd545d2b6a2d9181f4e5468 P 672a55e7d4874c5d99f3c5e3d9550f15: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240502 01:55:19.588830 25690 tablet_bootstrap.cc:492] T 75d71cdcf4d44353acb7d2f18696977b P 672a55e7d4874c5d99f3c5e3d9550f15: Bootstrap starting.
I20240502 01:55:19.589051 25689 raft_consensus.cc:348] T 450538b8217d40ce945d51e2c171ee36 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } }
I20240502 01:55:19.589075 25688 raft_consensus.cc:348] T 75d71cdcf4d44353acb7d2f18696977b P 96e5c2fd5d624c2da735e72b387a12d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } }
I20240502 01:55:19.589284 25688 raft_consensus.cc:374] T 75d71cdcf4d44353acb7d2f18696977b P 96e5c2fd5d624c2da735e72b387a12d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:19.589320 25689 raft_consensus.cc:374] T 450538b8217d40ce945d51e2c171ee36 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:19.589393 25689 raft_consensus.cc:729] T 450538b8217d40ce945d51e2c171ee36 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75bd3f0e8d7c4e8aa92a353301afeb21, State: Initialized, Role: FOLLOWER
I20240502 01:55:19.589675 25689 consensus_queue.cc:260] T 450538b8217d40ce945d51e2c171ee36 P 75bd3f0e8d7c4e8aa92a353301afeb21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } }
I20240502 01:55:19.590291 25689 ts_tablet_manager.cc:1398] T 450538b8217d40ce945d51e2c171ee36 P 75bd3f0e8d7c4e8aa92a353301afeb21: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 01:55:19.590654 25689 tablet_bootstrap.cc:492] T b6beb3b47fd545d2b6a2d9181f4e5468 P 75bd3f0e8d7c4e8aa92a353301afeb21: Bootstrap starting.
I20240502 01:55:19.590919 25688 raft_consensus.cc:729] T 75d71cdcf4d44353acb7d2f18696977b P 96e5c2fd5d624c2da735e72b387a12d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96e5c2fd5d624c2da735e72b387a12d9, State: Initialized, Role: FOLLOWER
I20240502 01:55:19.591166 25690 tablet_bootstrap.cc:654] T 75d71cdcf4d44353acb7d2f18696977b P 672a55e7d4874c5d99f3c5e3d9550f15: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:19.591250 25688 consensus_queue.cc:260] T 75d71cdcf4d44353acb7d2f18696977b P 96e5c2fd5d624c2da735e72b387a12d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } }
I20240502 01:55:19.592806 25689 tablet_bootstrap.cc:654] T b6beb3b47fd545d2b6a2d9181f4e5468 P 75bd3f0e8d7c4e8aa92a353301afeb21: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:19.599422 25688 ts_tablet_manager.cc:1398] T 75d71cdcf4d44353acb7d2f18696977b P 96e5c2fd5d624c2da735e72b387a12d9: Time spent starting tablet: real 0.011s	user 0.004s	sys 0.004s
I20240502 01:55:19.600317 25688 tablet_bootstrap.cc:492] T b6beb3b47fd545d2b6a2d9181f4e5468 P 96e5c2fd5d624c2da735e72b387a12d9: Bootstrap starting.
I20240502 01:55:19.600693 25689 tablet_bootstrap.cc:492] T b6beb3b47fd545d2b6a2d9181f4e5468 P 75bd3f0e8d7c4e8aa92a353301afeb21: No bootstrap required, opened a new log
I20240502 01:55:19.600889 25689 ts_tablet_manager.cc:1367] T b6beb3b47fd545d2b6a2d9181f4e5468 P 75bd3f0e8d7c4e8aa92a353301afeb21: Time spent bootstrapping tablet: real 0.010s	user 0.004s	sys 0.004s
I20240502 01:55:19.602198 25690 tablet_bootstrap.cc:492] T 75d71cdcf4d44353acb7d2f18696977b P 672a55e7d4874c5d99f3c5e3d9550f15: No bootstrap required, opened a new log
I20240502 01:55:19.602257 25688 tablet_bootstrap.cc:654] T b6beb3b47fd545d2b6a2d9181f4e5468 P 96e5c2fd5d624c2da735e72b387a12d9: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:19.602383 25690 ts_tablet_manager.cc:1367] T 75d71cdcf4d44353acb7d2f18696977b P 672a55e7d4874c5d99f3c5e3d9550f15: Time spent bootstrapping tablet: real 0.014s	user 0.004s	sys 0.005s
I20240502 01:55:19.601727 25689 raft_consensus.cc:348] T b6beb3b47fd545d2b6a2d9181f4e5468 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } }
I20240502 01:55:19.602675 25689 raft_consensus.cc:374] T b6beb3b47fd545d2b6a2d9181f4e5468 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:19.602849 25689 raft_consensus.cc:729] T b6beb3b47fd545d2b6a2d9181f4e5468 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75bd3f0e8d7c4e8aa92a353301afeb21, State: Initialized, Role: FOLLOWER
I20240502 01:55:19.603251 25689 consensus_queue.cc:260] T b6beb3b47fd545d2b6a2d9181f4e5468 P 75bd3f0e8d7c4e8aa92a353301afeb21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } }
I20240502 01:55:19.603343 25690 raft_consensus.cc:348] T 75d71cdcf4d44353acb7d2f18696977b P 672a55e7d4874c5d99f3c5e3d9550f15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } }
I20240502 01:55:19.603729 25690 raft_consensus.cc:374] T 75d71cdcf4d44353acb7d2f18696977b P 672a55e7d4874c5d99f3c5e3d9550f15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:19.603886 25690 raft_consensus.cc:729] T 75d71cdcf4d44353acb7d2f18696977b P 672a55e7d4874c5d99f3c5e3d9550f15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 672a55e7d4874c5d99f3c5e3d9550f15, State: Initialized, Role: FOLLOWER
I20240502 01:55:19.604300 25689 ts_tablet_manager.cc:1398] T b6beb3b47fd545d2b6a2d9181f4e5468 P 75bd3f0e8d7c4e8aa92a353301afeb21: Time spent starting tablet: real 0.003s	user 0.003s	sys 0.000s
I20240502 01:55:19.604316 25690 consensus_queue.cc:260] T 75d71cdcf4d44353acb7d2f18696977b P 672a55e7d4874c5d99f3c5e3d9550f15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } }
I20240502 01:55:19.605001 25690 ts_tablet_manager.cc:1398] T 75d71cdcf4d44353acb7d2f18696977b P 672a55e7d4874c5d99f3c5e3d9550f15: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 01:55:19.605885 25688 tablet_bootstrap.cc:492] T b6beb3b47fd545d2b6a2d9181f4e5468 P 96e5c2fd5d624c2da735e72b387a12d9: No bootstrap required, opened a new log
I20240502 01:55:19.606062 25688 ts_tablet_manager.cc:1367] T b6beb3b47fd545d2b6a2d9181f4e5468 P 96e5c2fd5d624c2da735e72b387a12d9: Time spent bootstrapping tablet: real 0.006s	user 0.005s	sys 0.000s
I20240502 01:55:19.606681 25688 raft_consensus.cc:348] T b6beb3b47fd545d2b6a2d9181f4e5468 P 96e5c2fd5d624c2da735e72b387a12d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } }
I20240502 01:55:19.606877 25688 raft_consensus.cc:374] T b6beb3b47fd545d2b6a2d9181f4e5468 P 96e5c2fd5d624c2da735e72b387a12d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:19.606987 25688 raft_consensus.cc:729] T b6beb3b47fd545d2b6a2d9181f4e5468 P 96e5c2fd5d624c2da735e72b387a12d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96e5c2fd5d624c2da735e72b387a12d9, State: Initialized, Role: FOLLOWER
I20240502 01:55:19.607304 25688 consensus_queue.cc:260] T b6beb3b47fd545d2b6a2d9181f4e5468 P 96e5c2fd5d624c2da735e72b387a12d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } }
I20240502 01:55:19.607914 25688 ts_tablet_manager.cc:1398] T b6beb3b47fd545d2b6a2d9181f4e5468 P 96e5c2fd5d624c2da735e72b387a12d9: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 01:55:19.608275 25688 tablet_bootstrap.cc:492] T 450538b8217d40ce945d51e2c171ee36 P 96e5c2fd5d624c2da735e72b387a12d9: Bootstrap starting.
I20240502 01:55:19.610190 25688 tablet_bootstrap.cc:654] T 450538b8217d40ce945d51e2c171ee36 P 96e5c2fd5d624c2da735e72b387a12d9: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:19.612574 25688 tablet_bootstrap.cc:492] T 450538b8217d40ce945d51e2c171ee36 P 96e5c2fd5d624c2da735e72b387a12d9: No bootstrap required, opened a new log
I20240502 01:55:19.612790 25688 ts_tablet_manager.cc:1367] T 450538b8217d40ce945d51e2c171ee36 P 96e5c2fd5d624c2da735e72b387a12d9: Time spent bootstrapping tablet: real 0.005s	user 0.003s	sys 0.000s
I20240502 01:55:19.613523 25688 raft_consensus.cc:348] T 450538b8217d40ce945d51e2c171ee36 P 96e5c2fd5d624c2da735e72b387a12d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } }
I20240502 01:55:19.613723 25688 raft_consensus.cc:374] T 450538b8217d40ce945d51e2c171ee36 P 96e5c2fd5d624c2da735e72b387a12d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:19.613804 25688 raft_consensus.cc:729] T 450538b8217d40ce945d51e2c171ee36 P 96e5c2fd5d624c2da735e72b387a12d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96e5c2fd5d624c2da735e72b387a12d9, State: Initialized, Role: FOLLOWER
I20240502 01:55:19.614045 25688 consensus_queue.cc:260] T 450538b8217d40ce945d51e2c171ee36 P 96e5c2fd5d624c2da735e72b387a12d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } }
I20240502 01:55:19.614562 25688 ts_tablet_manager.cc:1398] T 450538b8217d40ce945d51e2c171ee36 P 96e5c2fd5d624c2da735e72b387a12d9: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 01:55:19.629276 25696 raft_consensus.cc:482] T 75d71cdcf4d44353acb7d2f18696977b P 96e5c2fd5d624c2da735e72b387a12d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:55:19.629518 25696 raft_consensus.cc:504] T 75d71cdcf4d44353acb7d2f18696977b P 96e5c2fd5d624c2da735e72b387a12d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } }
I20240502 01:55:19.631143 25696 leader_election.cc:288] T 75d71cdcf4d44353acb7d2f18696977b P 96e5c2fd5d624c2da735e72b387a12d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 75bd3f0e8d7c4e8aa92a353301afeb21 (127.9.176.194:41625), 672a55e7d4874c5d99f3c5e3d9550f15 (127.9.176.195:46579)
I20240502 01:55:19.638309 25002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75d71cdcf4d44353acb7d2f18696977b" candidate_uuid: "96e5c2fd5d624c2da735e72b387a12d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" is_pre_election: true
I20240502 01:55:19.638513 25134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75d71cdcf4d44353acb7d2f18696977b" candidate_uuid: "96e5c2fd5d624c2da735e72b387a12d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" is_pre_election: true
I20240502 01:55:19.638648 25002 raft_consensus.cc:2454] T 75d71cdcf4d44353acb7d2f18696977b P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96e5c2fd5d624c2da735e72b387a12d9 in term 0.
I20240502 01:55:19.638767 25134 raft_consensus.cc:2454] T 75d71cdcf4d44353acb7d2f18696977b P 672a55e7d4874c5d99f3c5e3d9550f15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96e5c2fd5d624c2da735e72b387a12d9 in term 0.
I20240502 01:55:19.639144 24803 leader_election.cc:302] T 75d71cdcf4d44353acb7d2f18696977b P 96e5c2fd5d624c2da735e72b387a12d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75bd3f0e8d7c4e8aa92a353301afeb21, 96e5c2fd5d624c2da735e72b387a12d9; no voters: 
I20240502 01:55:19.639530 25696 raft_consensus.cc:2781] T 75d71cdcf4d44353acb7d2f18696977b P 96e5c2fd5d624c2da735e72b387a12d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:55:19.639685 25696 raft_consensus.cc:482] T 75d71cdcf4d44353acb7d2f18696977b P 96e5c2fd5d624c2da735e72b387a12d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:55:19.639796 25696 raft_consensus.cc:3037] T 75d71cdcf4d44353acb7d2f18696977b P 96e5c2fd5d624c2da735e72b387a12d9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:19.641903 25696 raft_consensus.cc:504] T 75d71cdcf4d44353acb7d2f18696977b P 96e5c2fd5d624c2da735e72b387a12d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } }
I20240502 01:55:19.642549 25696 leader_election.cc:288] T 75d71cdcf4d44353acb7d2f18696977b P 96e5c2fd5d624c2da735e72b387a12d9 [CANDIDATE]: Term 1 election: Requested vote from peers 75bd3f0e8d7c4e8aa92a353301afeb21 (127.9.176.194:41625), 672a55e7d4874c5d99f3c5e3d9550f15 (127.9.176.195:46579)
I20240502 01:55:19.643024 25002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75d71cdcf4d44353acb7d2f18696977b" candidate_uuid: "96e5c2fd5d624c2da735e72b387a12d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21"
I20240502 01:55:19.643239 25002 raft_consensus.cc:3037] T 75d71cdcf4d44353acb7d2f18696977b P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:19.643172 25134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75d71cdcf4d44353acb7d2f18696977b" candidate_uuid: "96e5c2fd5d624c2da735e72b387a12d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "672a55e7d4874c5d99f3c5e3d9550f15"
I20240502 01:55:19.643417 25134 raft_consensus.cc:3037] T 75d71cdcf4d44353acb7d2f18696977b P 672a55e7d4874c5d99f3c5e3d9550f15 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:19.645262 25002 raft_consensus.cc:2454] T 75d71cdcf4d44353acb7d2f18696977b P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96e5c2fd5d624c2da735e72b387a12d9 in term 1.
I20240502 01:55:19.645421 25134 raft_consensus.cc:2454] T 75d71cdcf4d44353acb7d2f18696977b P 672a55e7d4874c5d99f3c5e3d9550f15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96e5c2fd5d624c2da735e72b387a12d9 in term 1.
I20240502 01:55:19.645768 24803 leader_election.cc:302] T 75d71cdcf4d44353acb7d2f18696977b P 96e5c2fd5d624c2da735e72b387a12d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75bd3f0e8d7c4e8aa92a353301afeb21, 96e5c2fd5d624c2da735e72b387a12d9; no voters: 
I20240502 01:55:19.646113 25696 raft_consensus.cc:2781] T 75d71cdcf4d44353acb7d2f18696977b P 96e5c2fd5d624c2da735e72b387a12d9 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:55:19.646577 25696 raft_consensus.cc:686] T 75d71cdcf4d44353acb7d2f18696977b P 96e5c2fd5d624c2da735e72b387a12d9 [term 1 LEADER]: Becoming Leader. State: Replica: 96e5c2fd5d624c2da735e72b387a12d9, State: Running, Role: LEADER
I20240502 01:55:19.647073 25696 consensus_queue.cc:237] T 75d71cdcf4d44353acb7d2f18696977b P 96e5c2fd5d624c2da735e72b387a12d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } }
I20240502 01:55:19.650859 24726 catalog_manager.cc:5484] T 75d71cdcf4d44353acb7d2f18696977b P 96e5c2fd5d624c2da735e72b387a12d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 96e5c2fd5d624c2da735e72b387a12d9 (127.9.176.193). New cstate: current_term: 1 leader_uuid: "96e5c2fd5d624c2da735e72b387a12d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } health_report { overall_health: UNKNOWN } } }
I20240502 01:55:19.714908 25625 raft_consensus.cc:482] T 450538b8217d40ce945d51e2c171ee36 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:55:19.715157 25625 raft_consensus.cc:504] T 450538b8217d40ce945d51e2c171ee36 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } }
I20240502 01:55:19.716096 25625 leader_election.cc:288] T 450538b8217d40ce945d51e2c171ee36 P 672a55e7d4874c5d99f3c5e3d9550f15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96e5c2fd5d624c2da735e72b387a12d9 (127.9.176.193:32783), 75bd3f0e8d7c4e8aa92a353301afeb21 (127.9.176.194:41625)
I20240502 01:55:19.716922 25002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "450538b8217d40ce945d51e2c171ee36" candidate_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" is_pre_election: true
I20240502 01:55:19.716900 24869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "450538b8217d40ce945d51e2c171ee36" candidate_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96e5c2fd5d624c2da735e72b387a12d9" is_pre_election: true
I20240502 01:55:19.717168 25002 raft_consensus.cc:2454] T 450538b8217d40ce945d51e2c171ee36 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 672a55e7d4874c5d99f3c5e3d9550f15 in term 0.
I20240502 01:55:19.717190 24869 raft_consensus.cc:2454] T 450538b8217d40ce945d51e2c171ee36 P 96e5c2fd5d624c2da735e72b387a12d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 672a55e7d4874c5d99f3c5e3d9550f15 in term 0.
I20240502 01:55:19.717690 25068 leader_election.cc:302] T 450538b8217d40ce945d51e2c171ee36 P 672a55e7d4874c5d99f3c5e3d9550f15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 672a55e7d4874c5d99f3c5e3d9550f15, 96e5c2fd5d624c2da735e72b387a12d9; no voters: 
I20240502 01:55:19.718014 25625 raft_consensus.cc:2781] T 450538b8217d40ce945d51e2c171ee36 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:55:19.718128 25625 raft_consensus.cc:482] T 450538b8217d40ce945d51e2c171ee36 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:55:19.718225 25625 raft_consensus.cc:3037] T 450538b8217d40ce945d51e2c171ee36 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:19.720196 25625 raft_consensus.cc:504] T 450538b8217d40ce945d51e2c171ee36 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } }
I20240502 01:55:19.720762 25625 leader_election.cc:288] T 450538b8217d40ce945d51e2c171ee36 P 672a55e7d4874c5d99f3c5e3d9550f15 [CANDIDATE]: Term 1 election: Requested vote from peers 96e5c2fd5d624c2da735e72b387a12d9 (127.9.176.193:32783), 75bd3f0e8d7c4e8aa92a353301afeb21 (127.9.176.194:41625)
I20240502 01:55:19.721338 24869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "450538b8217d40ce945d51e2c171ee36" candidate_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96e5c2fd5d624c2da735e72b387a12d9"
I20240502 01:55:19.721359 25002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "450538b8217d40ce945d51e2c171ee36" candidate_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21"
I20240502 01:55:19.721566 25002 raft_consensus.cc:3037] T 450538b8217d40ce945d51e2c171ee36 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:19.721589 24869 raft_consensus.cc:3037] T 450538b8217d40ce945d51e2c171ee36 P 96e5c2fd5d624c2da735e72b387a12d9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:19.724442 24869 raft_consensus.cc:2454] T 450538b8217d40ce945d51e2c171ee36 P 96e5c2fd5d624c2da735e72b387a12d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 672a55e7d4874c5d99f3c5e3d9550f15 in term 1.
I20240502 01:55:19.724462 25002 raft_consensus.cc:2454] T 450538b8217d40ce945d51e2c171ee36 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 672a55e7d4874c5d99f3c5e3d9550f15 in term 1.
I20240502 01:55:19.724921 25068 leader_election.cc:302] T 450538b8217d40ce945d51e2c171ee36 P 672a55e7d4874c5d99f3c5e3d9550f15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 672a55e7d4874c5d99f3c5e3d9550f15, 96e5c2fd5d624c2da735e72b387a12d9; no voters: 
I20240502 01:55:19.725198 25625 raft_consensus.cc:2781] T 450538b8217d40ce945d51e2c171ee36 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:55:19.725376 25625 raft_consensus.cc:686] T 450538b8217d40ce945d51e2c171ee36 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 1 LEADER]: Becoming Leader. State: Replica: 672a55e7d4874c5d99f3c5e3d9550f15, State: Running, Role: LEADER
I20240502 01:55:19.725672 25625 consensus_queue.cc:237] T 450538b8217d40ce945d51e2c171ee36 P 672a55e7d4874c5d99f3c5e3d9550f15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } }
I20240502 01:55:19.728610 24726 catalog_manager.cc:5484] T 450538b8217d40ce945d51e2c171ee36 P 672a55e7d4874c5d99f3c5e3d9550f15 reported cstate change: term changed from 0 to 1, leader changed from <none> to 672a55e7d4874c5d99f3c5e3d9550f15 (127.9.176.195). New cstate: current_term: 1 leader_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } health_report { overall_health: HEALTHY } } }
I20240502 01:55:19.751732 25625 raft_consensus.cc:482] T b6beb3b47fd545d2b6a2d9181f4e5468 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:55:19.751965 25625 raft_consensus.cc:504] T b6beb3b47fd545d2b6a2d9181f4e5468 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } }
I20240502 01:55:19.752682 25625 leader_election.cc:288] T b6beb3b47fd545d2b6a2d9181f4e5468 P 672a55e7d4874c5d99f3c5e3d9550f15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96e5c2fd5d624c2da735e72b387a12d9 (127.9.176.193:32783), 75bd3f0e8d7c4e8aa92a353301afeb21 (127.9.176.194:41625)
I20240502 01:55:19.753327 24869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6beb3b47fd545d2b6a2d9181f4e5468" candidate_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96e5c2fd5d624c2da735e72b387a12d9" is_pre_election: true
I20240502 01:55:19.753383 25002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6beb3b47fd545d2b6a2d9181f4e5468" candidate_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" is_pre_election: true
I20240502 01:55:19.753540 24869 raft_consensus.cc:2454] T b6beb3b47fd545d2b6a2d9181f4e5468 P 96e5c2fd5d624c2da735e72b387a12d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 672a55e7d4874c5d99f3c5e3d9550f15 in term 0.
I20240502 01:55:19.753672 25002 raft_consensus.cc:2454] T b6beb3b47fd545d2b6a2d9181f4e5468 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 672a55e7d4874c5d99f3c5e3d9550f15 in term 0.
I20240502 01:55:19.754042 25068 leader_election.cc:302] T b6beb3b47fd545d2b6a2d9181f4e5468 P 672a55e7d4874c5d99f3c5e3d9550f15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 672a55e7d4874c5d99f3c5e3d9550f15, 96e5c2fd5d624c2da735e72b387a12d9; no voters: 
I20240502 01:55:19.754315 25625 raft_consensus.cc:2781] T b6beb3b47fd545d2b6a2d9181f4e5468 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:55:19.754442 25625 raft_consensus.cc:482] T b6beb3b47fd545d2b6a2d9181f4e5468 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:55:19.754535 25625 raft_consensus.cc:3037] T b6beb3b47fd545d2b6a2d9181f4e5468 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:19.756913 25625 raft_consensus.cc:504] T b6beb3b47fd545d2b6a2d9181f4e5468 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } }
I20240502 01:55:19.757467 25625 leader_election.cc:288] T b6beb3b47fd545d2b6a2d9181f4e5468 P 672a55e7d4874c5d99f3c5e3d9550f15 [CANDIDATE]: Term 1 election: Requested vote from peers 96e5c2fd5d624c2da735e72b387a12d9 (127.9.176.193:32783), 75bd3f0e8d7c4e8aa92a353301afeb21 (127.9.176.194:41625)
I20240502 01:55:19.757941 24869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6beb3b47fd545d2b6a2d9181f4e5468" candidate_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96e5c2fd5d624c2da735e72b387a12d9"
I20240502 01:55:19.757982 25002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6beb3b47fd545d2b6a2d9181f4e5468" candidate_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21"
I20240502 01:55:19.758129 24869 raft_consensus.cc:3037] T b6beb3b47fd545d2b6a2d9181f4e5468 P 96e5c2fd5d624c2da735e72b387a12d9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:19.758194 25002 raft_consensus.cc:3037] T b6beb3b47fd545d2b6a2d9181f4e5468 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:19.760470 24869 raft_consensus.cc:2454] T b6beb3b47fd545d2b6a2d9181f4e5468 P 96e5c2fd5d624c2da735e72b387a12d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 672a55e7d4874c5d99f3c5e3d9550f15 in term 1.
I20240502 01:55:19.760632 25002 raft_consensus.cc:2454] T b6beb3b47fd545d2b6a2d9181f4e5468 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 672a55e7d4874c5d99f3c5e3d9550f15 in term 1.
I20240502 01:55:19.761101 25068 leader_election.cc:302] T b6beb3b47fd545d2b6a2d9181f4e5468 P 672a55e7d4874c5d99f3c5e3d9550f15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 672a55e7d4874c5d99f3c5e3d9550f15, 96e5c2fd5d624c2da735e72b387a12d9; no voters: 
I20240502 01:55:19.761412 25625 raft_consensus.cc:2781] T b6beb3b47fd545d2b6a2d9181f4e5468 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:55:19.761566 25625 raft_consensus.cc:686] T b6beb3b47fd545d2b6a2d9181f4e5468 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 1 LEADER]: Becoming Leader. State: Replica: 672a55e7d4874c5d99f3c5e3d9550f15, State: Running, Role: LEADER
I20240502 01:55:19.761874 25625 consensus_queue.cc:237] T b6beb3b47fd545d2b6a2d9181f4e5468 P 672a55e7d4874c5d99f3c5e3d9550f15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } }
I20240502 01:55:19.764782 24726 catalog_manager.cc:5484] T b6beb3b47fd545d2b6a2d9181f4e5468 P 672a55e7d4874c5d99f3c5e3d9550f15 reported cstate change: term changed from 0 to 1, leader changed from <none> to 672a55e7d4874c5d99f3c5e3d9550f15 (127.9.176.195). New cstate: current_term: 1 leader_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } health_report { overall_health: HEALTHY } } }
I20240502 01:55:20.105177 25625 consensus_queue.cc:1035] T 450538b8217d40ce945d51e2c171ee36 P 672a55e7d4874c5d99f3c5e3d9550f15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:20.115685 25625 consensus_queue.cc:1035] T 450538b8217d40ce945d51e2c171ee36 P 672a55e7d4874c5d99f3c5e3d9550f15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:20.139145 25696 consensus_queue.cc:1035] T 75d71cdcf4d44353acb7d2f18696977b P 96e5c2fd5d624c2da735e72b387a12d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:20.144874 25703 consensus_queue.cc:1035] T 75d71cdcf4d44353acb7d2f18696977b P 96e5c2fd5d624c2da735e72b387a12d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:55:20.184716 25707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:55:20.185106 25707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:55:20.193128 25707 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240502 01:55:20.232684 25637 consensus_queue.cc:1035] T b6beb3b47fd545d2b6a2d9181f4e5468 P 672a55e7d4874c5d99f3c5e3d9550f15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:20.239727 25711 consensus_queue.cc:1035] T b6beb3b47fd545d2b6a2d9181f4e5468 P 672a55e7d4874c5d99f3c5e3d9550f15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:55:20.486810 25734 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.9.176.195:46579, user_credentials={real_user=slave}} blocked reactor thread for 62547.3us

INSERT report
    rows total: 2000
    time total: 280.458 ms
  time per row: 0.140229 ms
I20240502 01:55:20.631062 24725 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53224:
name: "rebalance_test_table_2"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
W20240502 01:55:20.631896 24725 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 01:55:20.640275 24980 tablet_service.cc:1467] Processing CreateTablet for tablet 08bb5eb2c2814019b6eda6fbbc288245 (DEFAULT_TABLE table=rebalance_test_table_2 [id=26fb053e91c344cab36e6f0312de1a3c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:20.640393 24849 tablet_service.cc:1467] Processing CreateTablet for tablet 08bb5eb2c2814019b6eda6fbbc288245 (DEFAULT_TABLE table=rebalance_test_table_2 [id=26fb053e91c344cab36e6f0312de1a3c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:20.640797 24980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08bb5eb2c2814019b6eda6fbbc288245. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:20.640839 24849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08bb5eb2c2814019b6eda6fbbc288245. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:20.641431 25113 tablet_service.cc:1467] Processing CreateTablet for tablet 08bb5eb2c2814019b6eda6fbbc288245 (DEFAULT_TABLE table=rebalance_test_table_2 [id=26fb053e91c344cab36e6f0312de1a3c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:20.641858 25113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08bb5eb2c2814019b6eda6fbbc288245. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:20.641803 24848 tablet_service.cc:1467] Processing CreateTablet for tablet 32b0189cb19f42efbe7118003e9390f9 (DEFAULT_TABLE table=rebalance_test_table_2 [id=26fb053e91c344cab36e6f0312de1a3c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:20.642180 24848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32b0189cb19f42efbe7118003e9390f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:20.647751 24982 tablet_service.cc:1467] Processing CreateTablet for tablet 32b0189cb19f42efbe7118003e9390f9 (DEFAULT_TABLE table=rebalance_test_table_2 [id=26fb053e91c344cab36e6f0312de1a3c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:20.647929 25112 tablet_service.cc:1467] Processing CreateTablet for tablet 32b0189cb19f42efbe7118003e9390f9 (DEFAULT_TABLE table=rebalance_test_table_2 [id=26fb053e91c344cab36e6f0312de1a3c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:20.648173 24982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32b0189cb19f42efbe7118003e9390f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:20.648274 25112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32b0189cb19f42efbe7118003e9390f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:20.650655 25770 tablet_bootstrap.cc:492] T 08bb5eb2c2814019b6eda6fbbc288245 P 75bd3f0e8d7c4e8aa92a353301afeb21: Bootstrap starting.
I20240502 01:55:20.650664 25114 tablet_service.cc:1467] Processing CreateTablet for tablet a9c57322de524bce8a114d9a94bd019d (DEFAULT_TABLE table=rebalance_test_table_2 [id=26fb053e91c344cab36e6f0312de1a3c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:20.651034 25114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9c57322de524bce8a114d9a94bd019d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:20.655167 25770 tablet_bootstrap.cc:654] T 08bb5eb2c2814019b6eda6fbbc288245 P 75bd3f0e8d7c4e8aa92a353301afeb21: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:20.655853 24848 tablet_service.cc:1467] Processing CreateTablet for tablet a9c57322de524bce8a114d9a94bd019d (DEFAULT_TABLE table=rebalance_test_table_2 [id=26fb053e91c344cab36e6f0312de1a3c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:20.656329 24848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9c57322de524bce8a114d9a94bd019d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:20.666484 24982 tablet_service.cc:1467] Processing CreateTablet for tablet a9c57322de524bce8a114d9a94bd019d (DEFAULT_TABLE table=rebalance_test_table_2 [id=26fb053e91c344cab36e6f0312de1a3c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:20.666762 25773 tablet_bootstrap.cc:492] T a9c57322de524bce8a114d9a94bd019d P 672a55e7d4874c5d99f3c5e3d9550f15: Bootstrap starting.
I20240502 01:55:20.666913 25770 tablet_bootstrap.cc:492] T 08bb5eb2c2814019b6eda6fbbc288245 P 75bd3f0e8d7c4e8aa92a353301afeb21: No bootstrap required, opened a new log
I20240502 01:55:20.667097 24982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9c57322de524bce8a114d9a94bd019d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:20.667336 25770 ts_tablet_manager.cc:1367] T 08bb5eb2c2814019b6eda6fbbc288245 P 75bd3f0e8d7c4e8aa92a353301afeb21: Time spent bootstrapping tablet: real 0.017s	user 0.013s	sys 0.000s
I20240502 01:55:20.655882 25771 tablet_bootstrap.cc:492] T 08bb5eb2c2814019b6eda6fbbc288245 P 96e5c2fd5d624c2da735e72b387a12d9: Bootstrap starting.
I20240502 01:55:20.668619 25770 raft_consensus.cc:348] T 08bb5eb2c2814019b6eda6fbbc288245 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } }
I20240502 01:55:20.668936 25770 raft_consensus.cc:374] T 08bb5eb2c2814019b6eda6fbbc288245 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:20.669087 25770 raft_consensus.cc:729] T 08bb5eb2c2814019b6eda6fbbc288245 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75bd3f0e8d7c4e8aa92a353301afeb21, State: Initialized, Role: FOLLOWER
I20240502 01:55:20.669425 25770 consensus_queue.cc:260] T 08bb5eb2c2814019b6eda6fbbc288245 P 75bd3f0e8d7c4e8aa92a353301afeb21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } }
I20240502 01:55:20.669615 25773 tablet_bootstrap.cc:654] T a9c57322de524bce8a114d9a94bd019d P 672a55e7d4874c5d99f3c5e3d9550f15: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:20.670162 25770 ts_tablet_manager.cc:1398] T 08bb5eb2c2814019b6eda6fbbc288245 P 75bd3f0e8d7c4e8aa92a353301afeb21: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 01:55:20.670511 25770 tablet_bootstrap.cc:492] T 32b0189cb19f42efbe7118003e9390f9 P 75bd3f0e8d7c4e8aa92a353301afeb21: Bootstrap starting.
I20240502 01:55:20.671124 25771 tablet_bootstrap.cc:654] T 08bb5eb2c2814019b6eda6fbbc288245 P 96e5c2fd5d624c2da735e72b387a12d9: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:20.673347 25770 tablet_bootstrap.cc:654] T 32b0189cb19f42efbe7118003e9390f9 P 75bd3f0e8d7c4e8aa92a353301afeb21: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:20.680889 25771 tablet_bootstrap.cc:492] T 08bb5eb2c2814019b6eda6fbbc288245 P 96e5c2fd5d624c2da735e72b387a12d9: No bootstrap required, opened a new log
I20240502 01:55:20.681155 25771 ts_tablet_manager.cc:1367] T 08bb5eb2c2814019b6eda6fbbc288245 P 96e5c2fd5d624c2da735e72b387a12d9: Time spent bootstrapping tablet: real 0.025s	user 0.007s	sys 0.004s
I20240502 01:55:20.682154 25770 tablet_bootstrap.cc:492] T 32b0189cb19f42efbe7118003e9390f9 P 75bd3f0e8d7c4e8aa92a353301afeb21: No bootstrap required, opened a new log
I20240502 01:55:20.682350 25773 tablet_bootstrap.cc:492] T a9c57322de524bce8a114d9a94bd019d P 672a55e7d4874c5d99f3c5e3d9550f15: No bootstrap required, opened a new log
I20240502 01:55:20.682368 25770 ts_tablet_manager.cc:1367] T 32b0189cb19f42efbe7118003e9390f9 P 75bd3f0e8d7c4e8aa92a353301afeb21: Time spent bootstrapping tablet: real 0.012s	user 0.003s	sys 0.008s
I20240502 01:55:20.682538 25773 ts_tablet_manager.cc:1367] T a9c57322de524bce8a114d9a94bd019d P 672a55e7d4874c5d99f3c5e3d9550f15: Time spent bootstrapping tablet: real 0.016s	user 0.003s	sys 0.011s
I20240502 01:55:20.682484 25771 raft_consensus.cc:348] T 08bb5eb2c2814019b6eda6fbbc288245 P 96e5c2fd5d624c2da735e72b387a12d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } }
I20240502 01:55:20.682763 25771 raft_consensus.cc:374] T 08bb5eb2c2814019b6eda6fbbc288245 P 96e5c2fd5d624c2da735e72b387a12d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:20.682869 25771 raft_consensus.cc:729] T 08bb5eb2c2814019b6eda6fbbc288245 P 96e5c2fd5d624c2da735e72b387a12d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96e5c2fd5d624c2da735e72b387a12d9, State: Initialized, Role: FOLLOWER
I20240502 01:55:20.683146 25771 consensus_queue.cc:260] T 08bb5eb2c2814019b6eda6fbbc288245 P 96e5c2fd5d624c2da735e72b387a12d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } }
I20240502 01:55:20.683248 25770 raft_consensus.cc:348] T 32b0189cb19f42efbe7118003e9390f9 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } }
I20240502 01:55:20.683389 25773 raft_consensus.cc:348] T a9c57322de524bce8a114d9a94bd019d P 672a55e7d4874c5d99f3c5e3d9550f15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } }
I20240502 01:55:20.683620 25770 raft_consensus.cc:374] T 32b0189cb19f42efbe7118003e9390f9 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:20.683666 25773 raft_consensus.cc:374] T a9c57322de524bce8a114d9a94bd019d P 672a55e7d4874c5d99f3c5e3d9550f15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:20.683759 25770 raft_consensus.cc:729] T 32b0189cb19f42efbe7118003e9390f9 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75bd3f0e8d7c4e8aa92a353301afeb21, State: Initialized, Role: FOLLOWER
I20240502 01:55:20.683815 25773 raft_consensus.cc:729] T a9c57322de524bce8a114d9a94bd019d P 672a55e7d4874c5d99f3c5e3d9550f15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 672a55e7d4874c5d99f3c5e3d9550f15, State: Initialized, Role: FOLLOWER
I20240502 01:55:20.683888 25771 ts_tablet_manager.cc:1398] T 08bb5eb2c2814019b6eda6fbbc288245 P 96e5c2fd5d624c2da735e72b387a12d9: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.001s
I20240502 01:55:20.684252 25771 tablet_bootstrap.cc:492] T 32b0189cb19f42efbe7118003e9390f9 P 96e5c2fd5d624c2da735e72b387a12d9: Bootstrap starting.
I20240502 01:55:20.684113 25773 consensus_queue.cc:260] T a9c57322de524bce8a114d9a94bd019d P 672a55e7d4874c5d99f3c5e3d9550f15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } }
I20240502 01:55:20.684139 25770 consensus_queue.cc:260] T 32b0189cb19f42efbe7118003e9390f9 P 75bd3f0e8d7c4e8aa92a353301afeb21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } }
I20240502 01:55:20.684756 25773 ts_tablet_manager.cc:1398] T a9c57322de524bce8a114d9a94bd019d P 672a55e7d4874c5d99f3c5e3d9550f15: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.001s
I20240502 01:55:20.684978 25770 ts_tablet_manager.cc:1398] T 32b0189cb19f42efbe7118003e9390f9 P 75bd3f0e8d7c4e8aa92a353301afeb21: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.002s
I20240502 01:55:20.685097 25773 tablet_bootstrap.cc:492] T 08bb5eb2c2814019b6eda6fbbc288245 P 672a55e7d4874c5d99f3c5e3d9550f15: Bootstrap starting.
I20240502 01:55:20.685369 25770 tablet_bootstrap.cc:492] T a9c57322de524bce8a114d9a94bd019d P 75bd3f0e8d7c4e8aa92a353301afeb21: Bootstrap starting.
I20240502 01:55:20.686838 25773 tablet_bootstrap.cc:654] T 08bb5eb2c2814019b6eda6fbbc288245 P 672a55e7d4874c5d99f3c5e3d9550f15: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:20.687377 25771 tablet_bootstrap.cc:654] T 32b0189cb19f42efbe7118003e9390f9 P 96e5c2fd5d624c2da735e72b387a12d9: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:20.687376 25770 tablet_bootstrap.cc:654] T a9c57322de524bce8a114d9a94bd019d P 75bd3f0e8d7c4e8aa92a353301afeb21: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:20.694442 25771 tablet_bootstrap.cc:492] T 32b0189cb19f42efbe7118003e9390f9 P 96e5c2fd5d624c2da735e72b387a12d9: No bootstrap required, opened a new log
I20240502 01:55:20.694759 25771 ts_tablet_manager.cc:1367] T 32b0189cb19f42efbe7118003e9390f9 P 96e5c2fd5d624c2da735e72b387a12d9: Time spent bootstrapping tablet: real 0.011s	user 0.003s	sys 0.004s
I20240502 01:55:20.695763 25771 raft_consensus.cc:348] T 32b0189cb19f42efbe7118003e9390f9 P 96e5c2fd5d624c2da735e72b387a12d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } }
I20240502 01:55:20.696127 25771 raft_consensus.cc:374] T 32b0189cb19f42efbe7118003e9390f9 P 96e5c2fd5d624c2da735e72b387a12d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:20.696257 25771 raft_consensus.cc:729] T 32b0189cb19f42efbe7118003e9390f9 P 96e5c2fd5d624c2da735e72b387a12d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96e5c2fd5d624c2da735e72b387a12d9, State: Initialized, Role: FOLLOWER
I20240502 01:55:20.696653 25771 consensus_queue.cc:260] T 32b0189cb19f42efbe7118003e9390f9 P 96e5c2fd5d624c2da735e72b387a12d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } }
I20240502 01:55:20.698654 25771 ts_tablet_manager.cc:1398] T 32b0189cb19f42efbe7118003e9390f9 P 96e5c2fd5d624c2da735e72b387a12d9: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.000s
I20240502 01:55:20.699280 25771 tablet_bootstrap.cc:492] T a9c57322de524bce8a114d9a94bd019d P 96e5c2fd5d624c2da735e72b387a12d9: Bootstrap starting.
I20240502 01:55:20.700599 25770 tablet_bootstrap.cc:492] T a9c57322de524bce8a114d9a94bd019d P 75bd3f0e8d7c4e8aa92a353301afeb21: No bootstrap required, opened a new log
I20240502 01:55:20.700830 25770 ts_tablet_manager.cc:1367] T a9c57322de524bce8a114d9a94bd019d P 75bd3f0e8d7c4e8aa92a353301afeb21: Time spent bootstrapping tablet: real 0.016s	user 0.001s	sys 0.014s
I20240502 01:55:20.701426 25771 tablet_bootstrap.cc:654] T a9c57322de524bce8a114d9a94bd019d P 96e5c2fd5d624c2da735e72b387a12d9: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:20.701822 25770 raft_consensus.cc:348] T a9c57322de524bce8a114d9a94bd019d P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } }
I20240502 01:55:20.702116 25770 raft_consensus.cc:374] T a9c57322de524bce8a114d9a94bd019d P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:20.702252 25770 raft_consensus.cc:729] T a9c57322de524bce8a114d9a94bd019d P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75bd3f0e8d7c4e8aa92a353301afeb21, State: Initialized, Role: FOLLOWER
I20240502 01:55:20.702574 25770 consensus_queue.cc:260] T a9c57322de524bce8a114d9a94bd019d P 75bd3f0e8d7c4e8aa92a353301afeb21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } }
I20240502 01:55:20.703241 25770 ts_tablet_manager.cc:1398] T a9c57322de524bce8a114d9a94bd019d P 75bd3f0e8d7c4e8aa92a353301afeb21: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240502 01:55:20.703444 25773 tablet_bootstrap.cc:492] T 08bb5eb2c2814019b6eda6fbbc288245 P 672a55e7d4874c5d99f3c5e3d9550f15: No bootstrap required, opened a new log
I20240502 01:55:20.703667 25773 ts_tablet_manager.cc:1367] T 08bb5eb2c2814019b6eda6fbbc288245 P 672a55e7d4874c5d99f3c5e3d9550f15: Time spent bootstrapping tablet: real 0.019s	user 0.009s	sys 0.008s
I20240502 01:55:20.704572 25773 raft_consensus.cc:348] T 08bb5eb2c2814019b6eda6fbbc288245 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } }
I20240502 01:55:20.704828 25773 raft_consensus.cc:374] T 08bb5eb2c2814019b6eda6fbbc288245 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:20.704933 25773 raft_consensus.cc:729] T 08bb5eb2c2814019b6eda6fbbc288245 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 672a55e7d4874c5d99f3c5e3d9550f15, State: Initialized, Role: FOLLOWER
I20240502 01:55:20.705094 25771 tablet_bootstrap.cc:492] T a9c57322de524bce8a114d9a94bd019d P 96e5c2fd5d624c2da735e72b387a12d9: No bootstrap required, opened a new log
I20240502 01:55:20.705255 25771 ts_tablet_manager.cc:1367] T a9c57322de524bce8a114d9a94bd019d P 96e5c2fd5d624c2da735e72b387a12d9: Time spent bootstrapping tablet: real 0.006s	user 0.003s	sys 0.000s
I20240502 01:55:20.705225 25773 consensus_queue.cc:260] T 08bb5eb2c2814019b6eda6fbbc288245 P 672a55e7d4874c5d99f3c5e3d9550f15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } }
I20240502 01:55:20.706014 25773 ts_tablet_manager.cc:1398] T 08bb5eb2c2814019b6eda6fbbc288245 P 672a55e7d4874c5d99f3c5e3d9550f15: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 01:55:20.706483 25773 tablet_bootstrap.cc:492] T 32b0189cb19f42efbe7118003e9390f9 P 672a55e7d4874c5d99f3c5e3d9550f15: Bootstrap starting.
I20240502 01:55:20.706707 25771 raft_consensus.cc:348] T a9c57322de524bce8a114d9a94bd019d P 96e5c2fd5d624c2da735e72b387a12d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } }
I20240502 01:55:20.706988 25771 raft_consensus.cc:374] T a9c57322de524bce8a114d9a94bd019d P 96e5c2fd5d624c2da735e72b387a12d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:20.707152 25771 raft_consensus.cc:729] T a9c57322de524bce8a114d9a94bd019d P 96e5c2fd5d624c2da735e72b387a12d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96e5c2fd5d624c2da735e72b387a12d9, State: Initialized, Role: FOLLOWER
I20240502 01:55:20.707558 25771 consensus_queue.cc:260] T a9c57322de524bce8a114d9a94bd019d P 96e5c2fd5d624c2da735e72b387a12d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } }
I20240502 01:55:20.708315 25773 tablet_bootstrap.cc:654] T 32b0189cb19f42efbe7118003e9390f9 P 672a55e7d4874c5d99f3c5e3d9550f15: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:20.709066 25771 ts_tablet_manager.cc:1398] T a9c57322de524bce8a114d9a94bd019d P 96e5c2fd5d624c2da735e72b387a12d9: Time spent starting tablet: real 0.004s	user 0.003s	sys 0.000s
I20240502 01:55:20.711220 25773 tablet_bootstrap.cc:492] T 32b0189cb19f42efbe7118003e9390f9 P 672a55e7d4874c5d99f3c5e3d9550f15: No bootstrap required, opened a new log
I20240502 01:55:20.711398 25773 ts_tablet_manager.cc:1367] T 32b0189cb19f42efbe7118003e9390f9 P 672a55e7d4874c5d99f3c5e3d9550f15: Time spent bootstrapping tablet: real 0.005s	user 0.004s	sys 0.000s
I20240502 01:55:20.712342 25773 raft_consensus.cc:348] T 32b0189cb19f42efbe7118003e9390f9 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } }
I20240502 01:55:20.712587 25773 raft_consensus.cc:374] T 32b0189cb19f42efbe7118003e9390f9 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:20.712713 25773 raft_consensus.cc:729] T 32b0189cb19f42efbe7118003e9390f9 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 672a55e7d4874c5d99f3c5e3d9550f15, State: Initialized, Role: FOLLOWER
I20240502 01:55:20.713001 25773 consensus_queue.cc:260] T 32b0189cb19f42efbe7118003e9390f9 P 672a55e7d4874c5d99f3c5e3d9550f15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } }
I20240502 01:55:20.713603 25773 ts_tablet_manager.cc:1398] T 32b0189cb19f42efbe7118003e9390f9 P 672a55e7d4874c5d99f3c5e3d9550f15: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 01:55:20.714195 25620 raft_consensus.cc:482] T 08bb5eb2c2814019b6eda6fbbc288245 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:55:20.714417 25620 raft_consensus.cc:504] T 08bb5eb2c2814019b6eda6fbbc288245 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } }
I20240502 01:55:20.715029 25620 leader_election.cc:288] T 08bb5eb2c2814019b6eda6fbbc288245 P 75bd3f0e8d7c4e8aa92a353301afeb21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96e5c2fd5d624c2da735e72b387a12d9 (127.9.176.193:32783), 672a55e7d4874c5d99f3c5e3d9550f15 (127.9.176.195:46579)
I20240502 01:55:20.715529 24868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08bb5eb2c2814019b6eda6fbbc288245" candidate_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96e5c2fd5d624c2da735e72b387a12d9" is_pre_election: true
I20240502 01:55:20.715688 25134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08bb5eb2c2814019b6eda6fbbc288245" candidate_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" is_pre_election: true
I20240502 01:55:20.715858 24868 raft_consensus.cc:2454] T 08bb5eb2c2814019b6eda6fbbc288245 P 96e5c2fd5d624c2da735e72b387a12d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75bd3f0e8d7c4e8aa92a353301afeb21 in term 0.
I20240502 01:55:20.715927 25134 raft_consensus.cc:2454] T 08bb5eb2c2814019b6eda6fbbc288245 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75bd3f0e8d7c4e8aa92a353301afeb21 in term 0.
I20240502 01:55:20.716360 24935 leader_election.cc:302] T 08bb5eb2c2814019b6eda6fbbc288245 P 75bd3f0e8d7c4e8aa92a353301afeb21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75bd3f0e8d7c4e8aa92a353301afeb21, 96e5c2fd5d624c2da735e72b387a12d9; no voters: 
I20240502 01:55:20.716710 25620 raft_consensus.cc:2781] T 08bb5eb2c2814019b6eda6fbbc288245 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:55:20.716846 25620 raft_consensus.cc:482] T 08bb5eb2c2814019b6eda6fbbc288245 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:55:20.716969 25620 raft_consensus.cc:3037] T 08bb5eb2c2814019b6eda6fbbc288245 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:20.719262 25620 raft_consensus.cc:504] T 08bb5eb2c2814019b6eda6fbbc288245 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } }
I20240502 01:55:20.719888 25620 leader_election.cc:288] T 08bb5eb2c2814019b6eda6fbbc288245 P 75bd3f0e8d7c4e8aa92a353301afeb21 [CANDIDATE]: Term 1 election: Requested vote from peers 96e5c2fd5d624c2da735e72b387a12d9 (127.9.176.193:32783), 672a55e7d4874c5d99f3c5e3d9550f15 (127.9.176.195:46579)
I20240502 01:55:20.720386 24868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08bb5eb2c2814019b6eda6fbbc288245" candidate_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96e5c2fd5d624c2da735e72b387a12d9"
I20240502 01:55:20.720541 25134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08bb5eb2c2814019b6eda6fbbc288245" candidate_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "672a55e7d4874c5d99f3c5e3d9550f15"
I20240502 01:55:20.720659 24868 raft_consensus.cc:3037] T 08bb5eb2c2814019b6eda6fbbc288245 P 96e5c2fd5d624c2da735e72b387a12d9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:20.720791 25134 raft_consensus.cc:3037] T 08bb5eb2c2814019b6eda6fbbc288245 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:20.723686 25134 raft_consensus.cc:2454] T 08bb5eb2c2814019b6eda6fbbc288245 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75bd3f0e8d7c4e8aa92a353301afeb21 in term 1.
I20240502 01:55:20.723690 24868 raft_consensus.cc:2454] T 08bb5eb2c2814019b6eda6fbbc288245 P 96e5c2fd5d624c2da735e72b387a12d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75bd3f0e8d7c4e8aa92a353301afeb21 in term 1.
I20240502 01:55:20.724212 24935 leader_election.cc:302] T 08bb5eb2c2814019b6eda6fbbc288245 P 75bd3f0e8d7c4e8aa92a353301afeb21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 672a55e7d4874c5d99f3c5e3d9550f15, 75bd3f0e8d7c4e8aa92a353301afeb21; no voters: 
I20240502 01:55:20.724514 25620 raft_consensus.cc:2781] T 08bb5eb2c2814019b6eda6fbbc288245 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:55:20.724961 25620 raft_consensus.cc:686] T 08bb5eb2c2814019b6eda6fbbc288245 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 1 LEADER]: Becoming Leader. State: Replica: 75bd3f0e8d7c4e8aa92a353301afeb21, State: Running, Role: LEADER
I20240502 01:55:20.725258 25620 consensus_queue.cc:237] T 08bb5eb2c2814019b6eda6fbbc288245 P 75bd3f0e8d7c4e8aa92a353301afeb21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } }
I20240502 01:55:20.728423 24721 catalog_manager.cc:5484] T 08bb5eb2c2814019b6eda6fbbc288245 P 75bd3f0e8d7c4e8aa92a353301afeb21 reported cstate change: term changed from 0 to 1, leader changed from <none> to 75bd3f0e8d7c4e8aa92a353301afeb21 (127.9.176.194). New cstate: current_term: 1 leader_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } health_report { overall_health: UNKNOWN } } }
I20240502 01:55:20.758620 25752 raft_consensus.cc:482] T 32b0189cb19f42efbe7118003e9390f9 P 96e5c2fd5d624c2da735e72b387a12d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:55:20.758834 25752 raft_consensus.cc:504] T 32b0189cb19f42efbe7118003e9390f9 P 96e5c2fd5d624c2da735e72b387a12d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } }
I20240502 01:55:20.759778 25752 leader_election.cc:288] T 32b0189cb19f42efbe7118003e9390f9 P 96e5c2fd5d624c2da735e72b387a12d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 75bd3f0e8d7c4e8aa92a353301afeb21 (127.9.176.194:41625), 672a55e7d4874c5d99f3c5e3d9550f15 (127.9.176.195:46579)
I20240502 01:55:20.761283 25002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32b0189cb19f42efbe7118003e9390f9" candidate_uuid: "96e5c2fd5d624c2da735e72b387a12d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" is_pre_election: true
I20240502 01:55:20.761587 25002 raft_consensus.cc:2454] T 32b0189cb19f42efbe7118003e9390f9 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96e5c2fd5d624c2da735e72b387a12d9 in term 0.
I20240502 01:55:20.761793 25134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32b0189cb19f42efbe7118003e9390f9" candidate_uuid: "96e5c2fd5d624c2da735e72b387a12d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" is_pre_election: true
I20240502 01:55:20.762238 24803 leader_election.cc:302] T 32b0189cb19f42efbe7118003e9390f9 P 96e5c2fd5d624c2da735e72b387a12d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75bd3f0e8d7c4e8aa92a353301afeb21, 96e5c2fd5d624c2da735e72b387a12d9; no voters: 
I20240502 01:55:20.762435 25134 raft_consensus.cc:2454] T 32b0189cb19f42efbe7118003e9390f9 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96e5c2fd5d624c2da735e72b387a12d9 in term 0.
I20240502 01:55:20.762614 25752 raft_consensus.cc:2781] T 32b0189cb19f42efbe7118003e9390f9 P 96e5c2fd5d624c2da735e72b387a12d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:55:20.762733 25752 raft_consensus.cc:482] T 32b0189cb19f42efbe7118003e9390f9 P 96e5c2fd5d624c2da735e72b387a12d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:55:20.762821 25752 raft_consensus.cc:3037] T 32b0189cb19f42efbe7118003e9390f9 P 96e5c2fd5d624c2da735e72b387a12d9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:20.765930 25752 raft_consensus.cc:504] T 32b0189cb19f42efbe7118003e9390f9 P 96e5c2fd5d624c2da735e72b387a12d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } }
I20240502 01:55:20.766716 25752 leader_election.cc:288] T 32b0189cb19f42efbe7118003e9390f9 P 96e5c2fd5d624c2da735e72b387a12d9 [CANDIDATE]: Term 1 election: Requested vote from peers 75bd3f0e8d7c4e8aa92a353301afeb21 (127.9.176.194:41625), 672a55e7d4874c5d99f3c5e3d9550f15 (127.9.176.195:46579)
I20240502 01:55:20.767367 25002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32b0189cb19f42efbe7118003e9390f9" candidate_uuid: "96e5c2fd5d624c2da735e72b387a12d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21"
I20240502 01:55:20.767637 25002 raft_consensus.cc:3037] T 32b0189cb19f42efbe7118003e9390f9 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:20.767773 25134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32b0189cb19f42efbe7118003e9390f9" candidate_uuid: "96e5c2fd5d624c2da735e72b387a12d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "672a55e7d4874c5d99f3c5e3d9550f15"
I20240502 01:55:20.768023 25134 raft_consensus.cc:3037] T 32b0189cb19f42efbe7118003e9390f9 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:20.770874 25002 raft_consensus.cc:2454] T 32b0189cb19f42efbe7118003e9390f9 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96e5c2fd5d624c2da735e72b387a12d9 in term 1.
I20240502 01:55:20.771364 25134 raft_consensus.cc:2454] T 32b0189cb19f42efbe7118003e9390f9 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96e5c2fd5d624c2da735e72b387a12d9 in term 1.
I20240502 01:55:20.771488 24803 leader_election.cc:302] T 32b0189cb19f42efbe7118003e9390f9 P 96e5c2fd5d624c2da735e72b387a12d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75bd3f0e8d7c4e8aa92a353301afeb21, 96e5c2fd5d624c2da735e72b387a12d9; no voters: 
I20240502 01:55:20.771986 25752 raft_consensus.cc:2781] T 32b0189cb19f42efbe7118003e9390f9 P 96e5c2fd5d624c2da735e72b387a12d9 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:55:20.772213 25752 raft_consensus.cc:686] T 32b0189cb19f42efbe7118003e9390f9 P 96e5c2fd5d624c2da735e72b387a12d9 [term 1 LEADER]: Becoming Leader. State: Replica: 96e5c2fd5d624c2da735e72b387a12d9, State: Running, Role: LEADER
I20240502 01:55:20.772635 25752 consensus_queue.cc:237] T 32b0189cb19f42efbe7118003e9390f9 P 96e5c2fd5d624c2da735e72b387a12d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } }
I20240502 01:55:20.776367 24721 catalog_manager.cc:5484] T 32b0189cb19f42efbe7118003e9390f9 P 96e5c2fd5d624c2da735e72b387a12d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 96e5c2fd5d624c2da735e72b387a12d9 (127.9.176.193). New cstate: current_term: 1 leader_uuid: "96e5c2fd5d624c2da735e72b387a12d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } health_report { overall_health: UNKNOWN } } }
I20240502 01:55:20.808203 25751 raft_consensus.cc:482] T a9c57322de524bce8a114d9a94bd019d P 672a55e7d4874c5d99f3c5e3d9550f15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:55:20.808408 25751 raft_consensus.cc:504] T a9c57322de524bce8a114d9a94bd019d P 672a55e7d4874c5d99f3c5e3d9550f15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } }
I20240502 01:55:20.809222 25751 leader_election.cc:288] T a9c57322de524bce8a114d9a94bd019d P 672a55e7d4874c5d99f3c5e3d9550f15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 75bd3f0e8d7c4e8aa92a353301afeb21 (127.9.176.194:41625), 96e5c2fd5d624c2da735e72b387a12d9 (127.9.176.193:32783)
I20240502 01:55:20.809734 25002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9c57322de524bce8a114d9a94bd019d" candidate_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" is_pre_election: true
I20240502 01:55:20.809844 24868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9c57322de524bce8a114d9a94bd019d" candidate_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96e5c2fd5d624c2da735e72b387a12d9" is_pre_election: true
I20240502 01:55:20.810002 25002 raft_consensus.cc:2454] T a9c57322de524bce8a114d9a94bd019d P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 672a55e7d4874c5d99f3c5e3d9550f15 in term 0.
I20240502 01:55:20.810082 24868 raft_consensus.cc:2454] T a9c57322de524bce8a114d9a94bd019d P 96e5c2fd5d624c2da735e72b387a12d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 672a55e7d4874c5d99f3c5e3d9550f15 in term 0.
I20240502 01:55:20.810508 25068 leader_election.cc:302] T a9c57322de524bce8a114d9a94bd019d P 672a55e7d4874c5d99f3c5e3d9550f15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 672a55e7d4874c5d99f3c5e3d9550f15, 96e5c2fd5d624c2da735e72b387a12d9; no voters: 
I20240502 01:55:20.810809 25751 raft_consensus.cc:2781] T a9c57322de524bce8a114d9a94bd019d P 672a55e7d4874c5d99f3c5e3d9550f15 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:55:20.810945 25751 raft_consensus.cc:482] T a9c57322de524bce8a114d9a94bd019d P 672a55e7d4874c5d99f3c5e3d9550f15 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:55:20.811041 25751 raft_consensus.cc:3037] T a9c57322de524bce8a114d9a94bd019d P 672a55e7d4874c5d99f3c5e3d9550f15 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:20.813979 25751 raft_consensus.cc:504] T a9c57322de524bce8a114d9a94bd019d P 672a55e7d4874c5d99f3c5e3d9550f15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } }
I20240502 01:55:20.814733 25751 leader_election.cc:288] T a9c57322de524bce8a114d9a94bd019d P 672a55e7d4874c5d99f3c5e3d9550f15 [CANDIDATE]: Term 1 election: Requested vote from peers 75bd3f0e8d7c4e8aa92a353301afeb21 (127.9.176.194:41625), 96e5c2fd5d624c2da735e72b387a12d9 (127.9.176.193:32783)
I20240502 01:55:20.815246 25002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9c57322de524bce8a114d9a94bd019d" candidate_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21"
I20240502 01:55:20.815363 24868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9c57322de524bce8a114d9a94bd019d" candidate_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96e5c2fd5d624c2da735e72b387a12d9"
I20240502 01:55:20.815549 25002 raft_consensus.cc:3037] T a9c57322de524bce8a114d9a94bd019d P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:20.815621 24868 raft_consensus.cc:3037] T a9c57322de524bce8a114d9a94bd019d P 96e5c2fd5d624c2da735e72b387a12d9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:20.818578 25002 raft_consensus.cc:2454] T a9c57322de524bce8a114d9a94bd019d P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 672a55e7d4874c5d99f3c5e3d9550f15 in term 1.
I20240502 01:55:20.818599 24868 raft_consensus.cc:2454] T a9c57322de524bce8a114d9a94bd019d P 96e5c2fd5d624c2da735e72b387a12d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 672a55e7d4874c5d99f3c5e3d9550f15 in term 1.
I20240502 01:55:20.819072 25068 leader_election.cc:302] T a9c57322de524bce8a114d9a94bd019d P 672a55e7d4874c5d99f3c5e3d9550f15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 672a55e7d4874c5d99f3c5e3d9550f15, 96e5c2fd5d624c2da735e72b387a12d9; no voters: 
I20240502 01:55:20.819420 25751 raft_consensus.cc:2781] T a9c57322de524bce8a114d9a94bd019d P 672a55e7d4874c5d99f3c5e3d9550f15 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:55:20.819692 25751 raft_consensus.cc:686] T a9c57322de524bce8a114d9a94bd019d P 672a55e7d4874c5d99f3c5e3d9550f15 [term 1 LEADER]: Becoming Leader. State: Replica: 672a55e7d4874c5d99f3c5e3d9550f15, State: Running, Role: LEADER
I20240502 01:55:20.819985 25751 consensus_queue.cc:237] T a9c57322de524bce8a114d9a94bd019d P 672a55e7d4874c5d99f3c5e3d9550f15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } }
I20240502 01:55:20.823280 24721 catalog_manager.cc:5484] T a9c57322de524bce8a114d9a94bd019d P 672a55e7d4874c5d99f3c5e3d9550f15 reported cstate change: term changed from 0 to 1, leader changed from <none> to 672a55e7d4874c5d99f3c5e3d9550f15 (127.9.176.195). New cstate: current_term: 1 leader_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } health_report { overall_health: HEALTHY } } }
I20240502 01:55:21.134699 25620 consensus_queue.cc:1035] T 08bb5eb2c2814019b6eda6fbbc288245 P 75bd3f0e8d7c4e8aa92a353301afeb21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:21.141103 25782 consensus_queue.cc:1035] T 08bb5eb2c2814019b6eda6fbbc288245 P 75bd3f0e8d7c4e8aa92a353301afeb21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:21.172295 25752 consensus_queue.cc:1035] T 32b0189cb19f42efbe7118003e9390f9 P 96e5c2fd5d624c2da735e72b387a12d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:21.178294 25752 consensus_queue.cc:1035] T 32b0189cb19f42efbe7118003e9390f9 P 96e5c2fd5d624c2da735e72b387a12d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:55:21.223291 25786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:55:21.223698 25786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:55:21.231360 25786 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240502 01:55:21.313804 25751 consensus_queue.cc:1035] T a9c57322de524bce8a114d9a94bd019d P 672a55e7d4874c5d99f3c5e3d9550f15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:21.324519 25751 consensus_queue.cc:1035] T a9c57322de524bce8a114d9a94bd019d P 672a55e7d4874c5d99f3c5e3d9550f15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 332.106 ms
  time per row: 0.166053 ms
I20240502 01:55:21.712472  9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
/tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.196:39355
--local_ip_for_outbound_sockets=127.9.176.196
--tserver_master_addrs=127.9.176.254:38303
--webserver_port=36127
--webserver_interface=127.9.176.196
--builtin_ntp_servers=127.9.176.212:33577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 01:55:22.077445 25851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:55:22.077772 25851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:55:22.077903 25851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:55:22.085748 25851 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:55:22.085907 25851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:55:22.086067 25851 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.196
I20240502 01:55:22.095922 25851 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:33577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.196:39355
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.9.176.196
--webserver_port=36127
--tserver_master_addrs=127.9.176.254:38303
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.196
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:55:22.097316 25851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:55:22.099155 25851 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:55:22.111650 25857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:22.112049 25859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:22.112718 25856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:55:22.113355 25851 server_base.cc:1034] running on GCE node
I20240502 01:55:22.113981 25851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:55:22.115087 25851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:55:22.116324 25851 hybrid_clock.cc:648] HybridClock initialized: now 1714614922116256 us; error 67 us; skew 500 ppm
I20240502 01:55:22.116683 25851 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:55:22.119038 25851 webserver.cc:460] Webserver started at http://127.9.176.196:36127/ using document root <none> and password file <none>
I20240502 01:55:22.119877 25851 fs_manager.cc:359] Metadata directory not provided
I20240502 01:55:22.120026 25851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:55:22.126215 25851 fs_manager.cc:709] Time spent opening directory manager: real 0.004s	user 0.005s	sys 0.000s
I20240502 01:55:22.129213 25865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:55:22.130674 25851 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.000s
I20240502 01:55:22.130853 25851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "30921110704f47f88d68a7d9364d42cc"
format_stamp: "Formatted at 2024-05-02 01:55:12 on dist-test-slave-j3hh"
I20240502 01:55:22.131822 25851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:55:22.153910 25851 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:55:22.154843 25851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:55:22.155108 25851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:55:22.156073 25851 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:55:22.157585 25851 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:55:22.157658 25851 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:55:22.157744 25851 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:55:22.157788 25851 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:55:22.191135 25851 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.196:39355
I20240502 01:55:22.191272 25978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.196:39355 every 8 connection(s)
I20240502 01:55:22.192746 25851 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 01:55:22.197141  9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu as pid 25851
I20240502 01:55:22.197981  9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
/tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.198:34317
--local_ip_for_outbound_sockets=127.9.176.198
--tserver_master_addrs=127.9.176.254:38303
--webserver_port=43655
--webserver_interface=127.9.176.198
--builtin_ntp_servers=127.9.176.212:33577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:55:22.208506 25979 heartbeater.cc:347] Connected to a master server at 127.9.176.254:38303
I20240502 01:55:22.208878 25979 heartbeater.cc:464] Registering TS with master...
I20240502 01:55:22.209978 25979 heartbeater.cc:511] Master 127.9.176.254:38303 requested a full tablet report, sending...
I20240502 01:55:22.212198 24722 ts_manager.cc:194] Re-registered known tserver with Master: 30921110704f47f88d68a7d9364d42cc (127.9.176.196:39355)
I20240502 01:55:22.213573 24722 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.176.196:48883
W20240502 01:55:22.576404 25983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:55:22.576833 25983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:55:22.576951 25983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:55:22.585220 25983 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:55:22.585373 25983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:55:22.585500 25983 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.198
I20240502 01:55:22.595187 25983 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:33577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.198:34317
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.9.176.198
--webserver_port=43655
--tserver_master_addrs=127.9.176.254:38303
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.198
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:55:22.596604 25983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:55:22.598527 25983 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:55:22.612138 25988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:22.612552 25989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:22.613184 25991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:55:22.613881 25983 server_base.cc:1034] running on GCE node
I20240502 01:55:22.614521 25983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:55:22.615520 25983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:55:22.616793 25983 hybrid_clock.cc:648] HybridClock initialized: now 1714614922616737 us; error 49 us; skew 500 ppm
I20240502 01:55:22.617149 25983 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:55:22.619673 25983 webserver.cc:460] Webserver started at http://127.9.176.198:43655/ using document root <none> and password file <none>
I20240502 01:55:22.620472 25983 fs_manager.cc:359] Metadata directory not provided
I20240502 01:55:22.620622 25983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:55:22.626804 25983 fs_manager.cc:709] Time spent opening directory manager: real 0.004s	user 0.005s	sys 0.001s
I20240502 01:55:22.629875 25997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:55:22.631348 25983 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20240502 01:55:22.631539 25983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "d490b54f0e954ef59deb6750523f6027"
format_stamp: "Formatted at 2024-05-02 01:55:13 on dist-test-slave-j3hh"
I20240502 01:55:22.632498 25983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:55:22.646207 25983 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:55:22.647059 25983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:55:22.647338 25983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:55:22.648342 25983 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:55:22.649914 25983 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:55:22.649991 25983 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:55:22.650131 25983 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:55:22.650183 25983 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:55:22.686609 25983 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.198:34317
I20240502 01:55:22.686654 26111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.198:34317 every 8 connection(s)
I20240502 01:55:22.688141 25983 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 01:55:22.689229  9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu as pid 25983
I20240502 01:55:22.689914  9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
/tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.197:43479
--local_ip_for_outbound_sockets=127.9.176.197
--tserver_master_addrs=127.9.176.254:38303
--webserver_port=42303
--webserver_interface=127.9.176.197
--builtin_ntp_servers=127.9.176.212:33577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:55:22.702409 26112 heartbeater.cc:347] Connected to a master server at 127.9.176.254:38303
I20240502 01:55:22.702826 26112 heartbeater.cc:464] Registering TS with master...
I20240502 01:55:22.703986 26112 heartbeater.cc:511] Master 127.9.176.254:38303 requested a full tablet report, sending...
I20240502 01:55:22.705794 24725 ts_manager.cc:194] Re-registered known tserver with Master: d490b54f0e954ef59deb6750523f6027 (127.9.176.198:34317)
I20240502 01:55:22.706660 24725 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.176.198:50465
W20240502 01:55:23.061651 26115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:55:23.061970 26115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:55:23.062093 26115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:55:23.069859 26115 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:55:23.070001 26115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:55:23.070168 26115 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.197
I20240502 01:55:23.079802 26115 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:33577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.197:43479
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.9.176.197
--webserver_port=42303
--tserver_master_addrs=127.9.176.254:38303
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.197
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:55:23.081228 26115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:55:23.083165 26115 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:55:23.095789 26121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:23.096272 26120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:23.096722 26123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:55:23.097579 26115 server_base.cc:1034] running on GCE node
I20240502 01:55:23.098227 26115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:55:23.099239 26115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:55:23.100455 26115 hybrid_clock.cc:648] HybridClock initialized: now 1714614923100402 us; error 53 us; skew 500 ppm
I20240502 01:55:23.100797 26115 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:55:23.103091 26115 webserver.cc:460] Webserver started at http://127.9.176.197:42303/ using document root <none> and password file <none>
I20240502 01:55:23.103888 26115 fs_manager.cc:359] Metadata directory not provided
I20240502 01:55:23.104028 26115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:55:23.109588 26115 fs_manager.cc:709] Time spent opening directory manager: real 0.004s	user 0.005s	sys 0.000s
I20240502 01:55:23.112673 26129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:55:23.114210 26115 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.001s
I20240502 01:55:23.114427 26115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "6742b9db50c64ea394c2ca38ddee0872"
format_stamp: "Formatted at 2024-05-02 01:55:13 on dist-test-slave-j3hh"
I20240502 01:55:23.115360 26115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:55:23.144023 26115 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:55:23.144860 26115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:55:23.145135 26115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:55:23.146104 26115 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:55:23.147720 26115 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:55:23.147794 26115 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:55:23.147884 26115 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:55:23.147931 26115 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:55:23.182264 26115 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.197:43479
I20240502 01:55:23.182305 26242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.197:43479 every 8 connection(s)
I20240502 01:55:23.184212 26115 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 01:55:23.187304  9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu as pid 26115
I20240502 01:55:23.198119 26243 heartbeater.cc:347] Connected to a master server at 127.9.176.254:38303
I20240502 01:55:23.198465 26243 heartbeater.cc:464] Registering TS with master...
I20240502 01:55:23.199362 26243 heartbeater.cc:511] Master 127.9.176.254:38303 requested a full tablet report, sending...
I20240502 01:55:23.201058 24725 ts_manager.cc:194] Re-registered known tserver with Master: 6742b9db50c64ea394c2ca38ddee0872 (127.9.176.197:43479)
I20240502 01:55:23.202172 24725 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.176.197:52089
I20240502 01:55:23.216674 25979 heartbeater.cc:503] Master 127.9.176.254:38303 was elected leader, sending a full tablet report...
I20240502 01:55:23.709415 26112 heartbeater.cc:503] Master 127.9.176.254:38303 was elected leader, sending a full tablet report...
I20240502 01:55:23.715170 26177 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:23.715378 25914 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:23.731575 26045 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:23.738413 25112 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:55:23.751866 24982 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:55:23.767843 24848 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:55:23.932556 25914 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:23.934194 25112 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:55:23.946849 26177 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:23.955521 24982 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:55:23.980044 24848 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:55:24.003702 26045 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:24.189812 25914 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:24.205520 26243 heartbeater.cc:503] Master 127.9.176.254:38303 was elected leader, sending a full tablet report...
I20240502 01:55:24.209988 26177 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:24.212978 24982 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:55:24.229610 25112 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:55:24.231452 26045 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:24.247116 24848 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:55:24.401744 25914 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:24.411761 25112 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:55:24.421348 26177 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:24.442997 24982 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:55:24.456513 24848 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:55:24.457302 26045 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:24.546880 25002 consensus_queue.cc:237] T 08bb5eb2c2814019b6eda6fbbc288245 P 75bd3f0e8d7c4e8aa92a353301afeb21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } attrs { replace: true } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: true } }
I20240502 01:55:24.550307 24868 raft_consensus.cc:1261] T 08bb5eb2c2814019b6eda6fbbc288245 P 96e5c2fd5d624c2da735e72b387a12d9 [term 1 FOLLOWER]: Refusing update from remote peer 75bd3f0e8d7c4e8aa92a353301afeb21: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:24.550769 25133 raft_consensus.cc:1261] T 08bb5eb2c2814019b6eda6fbbc288245 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 1 FOLLOWER]: Refusing update from remote peer 75bd3f0e8d7c4e8aa92a353301afeb21: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:24.551015 26395 consensus_queue.cc:1035] T 08bb5eb2c2814019b6eda6fbbc288245 P 75bd3f0e8d7c4e8aa92a353301afeb21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:24.551455 26396 consensus_queue.cc:1035] T 08bb5eb2c2814019b6eda6fbbc288245 P 75bd3f0e8d7c4e8aa92a353301afeb21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:24.555361 26395 raft_consensus.cc:2932] T 08bb5eb2c2814019b6eda6fbbc288245 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 30921110704f47f88d68a7d9364d42cc (127.9.176.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } attrs { replace: true } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: true } } }
I20240502 01:55:24.556088 25133 raft_consensus.cc:2932] T 08bb5eb2c2814019b6eda6fbbc288245 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 30921110704f47f88d68a7d9364d42cc (127.9.176.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } attrs { replace: true } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: true } } }
W20240502 01:55:24.559990 24937 consensus_peers.cc:485] T 08bb5eb2c2814019b6eda6fbbc288245 P 75bd3f0e8d7c4e8aa92a353301afeb21 -> Peer 30921110704f47f88d68a7d9364d42cc (127.9.176.196:39355): Couldn't send request to peer 30921110704f47f88d68a7d9364d42cc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 08bb5eb2c2814019b6eda6fbbc288245. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:24.562755 24722 catalog_manager.cc:5484] T 08bb5eb2c2814019b6eda6fbbc288245 P 672a55e7d4874c5d99f3c5e3d9550f15 reported cstate change: config changed from index -1 to 4, NON_VOTER 30921110704f47f88d68a7d9364d42cc (127.9.176.196) added. New cstate: current_term: 1 leader_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } attrs { replace: true } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: true } } }
I20240502 01:55:24.562991 24868 raft_consensus.cc:2932] T 08bb5eb2c2814019b6eda6fbbc288245 P 96e5c2fd5d624c2da735e72b387a12d9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 30921110704f47f88d68a7d9364d42cc (127.9.176.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } attrs { replace: true } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: true } } }
I20240502 01:55:24.582962 25133 consensus_queue.cc:237] T 450538b8217d40ce945d51e2c171ee36 P 672a55e7d4874c5d99f3c5e3d9550f15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { replace: true } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: true } }
I20240502 01:55:24.585891 24868 raft_consensus.cc:1261] T 450538b8217d40ce945d51e2c171ee36 P 96e5c2fd5d624c2da735e72b387a12d9 [term 1 FOLLOWER]: Refusing update from remote peer 672a55e7d4874c5d99f3c5e3d9550f15: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:24.586628 26319 consensus_queue.cc:1035] T 450538b8217d40ce945d51e2c171ee36 P 672a55e7d4874c5d99f3c5e3d9550f15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:24.588843 25002 raft_consensus.cc:1261] T 450538b8217d40ce945d51e2c171ee36 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 1 FOLLOWER]: Refusing update from remote peer 672a55e7d4874c5d99f3c5e3d9550f15: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:24.589674 26410 consensus_queue.cc:1035] T 450538b8217d40ce945d51e2c171ee36 P 672a55e7d4874c5d99f3c5e3d9550f15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:24.590540 26410 raft_consensus.cc:2932] T 450538b8217d40ce945d51e2c171ee36 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6742b9db50c64ea394c2ca38ddee0872 (127.9.176.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { replace: true } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: true } } }
I20240502 01:55:24.599231 24868 raft_consensus.cc:2932] T 450538b8217d40ce945d51e2c171ee36 P 96e5c2fd5d624c2da735e72b387a12d9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6742b9db50c64ea394c2ca38ddee0872 (127.9.176.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { replace: true } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: true } } }
I20240502 01:55:24.599650 24725 catalog_manager.cc:5484] T 450538b8217d40ce945d51e2c171ee36 P 672a55e7d4874c5d99f3c5e3d9550f15 reported cstate change: config changed from index -1 to 4, NON_VOTER 6742b9db50c64ea394c2ca38ddee0872 (127.9.176.197) added. New cstate: current_term: 1 leader_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 01:55:24.601174 25067 consensus_peers.cc:485] T 450538b8217d40ce945d51e2c171ee36 P 672a55e7d4874c5d99f3c5e3d9550f15 -> Peer 6742b9db50c64ea394c2ca38ddee0872 (127.9.176.197:43479): Couldn't send request to peer 6742b9db50c64ea394c2ca38ddee0872. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 450538b8217d40ce945d51e2c171ee36. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:24.602471 25002 raft_consensus.cc:2932] T 450538b8217d40ce945d51e2c171ee36 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6742b9db50c64ea394c2ca38ddee0872 (127.9.176.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { replace: true } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: true } } }
I20240502 01:55:24.617594 25002 consensus_queue.cc:237] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 75bd3f0e8d7c4e8aa92a353301afeb21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } attrs { replace: true } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: true } }
I20240502 01:55:24.620419 25133 raft_consensus.cc:1261] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 1 FOLLOWER]: Refusing update from remote peer 75bd3f0e8d7c4e8aa92a353301afeb21: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:24.620642 24868 raft_consensus.cc:1261] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 96e5c2fd5d624c2da735e72b387a12d9 [term 1 FOLLOWER]: Refusing update from remote peer 75bd3f0e8d7c4e8aa92a353301afeb21: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:24.621385 26395 consensus_queue.cc:1035] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 75bd3f0e8d7c4e8aa92a353301afeb21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:24.621760 26396 consensus_queue.cc:1035] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 75bd3f0e8d7c4e8aa92a353301afeb21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:24.625487 26395 raft_consensus.cc:2932] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d490b54f0e954ef59deb6750523f6027 (127.9.176.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } attrs { replace: true } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: true } } }
I20240502 01:55:24.628554 24868 raft_consensus.cc:2932] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 96e5c2fd5d624c2da735e72b387a12d9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d490b54f0e954ef59deb6750523f6027 (127.9.176.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } attrs { replace: true } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: true } } }
W20240502 01:55:24.629243 24936 consensus_peers.cc:485] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 75bd3f0e8d7c4e8aa92a353301afeb21 -> Peer d490b54f0e954ef59deb6750523f6027 (127.9.176.198:34317): Couldn't send request to peer d490b54f0e954ef59deb6750523f6027. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1b76f351e8644d9e8ff27e3f1a963ba2. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:24.631656 24724 catalog_manager.cc:5484] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 75bd3f0e8d7c4e8aa92a353301afeb21 reported cstate change: config changed from index -1 to 4, NON_VOTER d490b54f0e954ef59deb6750523f6027 (127.9.176.198) added. New cstate: current_term: 1 leader_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:55:24.633291 25133 raft_consensus.cc:2932] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d490b54f0e954ef59deb6750523f6027 (127.9.176.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } attrs { replace: true } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: true } } }
I20240502 01:55:24.648380 25002 consensus_queue.cc:237] T f77025fd71924964afafcd32c9ed6923 P 75bd3f0e8d7c4e8aa92a353301afeb21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } attrs { replace: true } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: true } }
I20240502 01:55:24.650933 25133 raft_consensus.cc:1261] T f77025fd71924964afafcd32c9ed6923 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 1 FOLLOWER]: Refusing update from remote peer 75bd3f0e8d7c4e8aa92a353301afeb21: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:24.651072 24868 raft_consensus.cc:1261] T f77025fd71924964afafcd32c9ed6923 P 96e5c2fd5d624c2da735e72b387a12d9 [term 1 FOLLOWER]: Refusing update from remote peer 75bd3f0e8d7c4e8aa92a353301afeb21: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 01:55:24.651229 24937 consensus_peers.cc:485] T f77025fd71924964afafcd32c9ed6923 P 75bd3f0e8d7c4e8aa92a353301afeb21 -> Peer 30921110704f47f88d68a7d9364d42cc (127.9.176.196:39355): Couldn't send request to peer 30921110704f47f88d68a7d9364d42cc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f77025fd71924964afafcd32c9ed6923. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:24.651659 25620 consensus_queue.cc:1035] T f77025fd71924964afafcd32c9ed6923 P 75bd3f0e8d7c4e8aa92a353301afeb21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:24.652033 26395 consensus_queue.cc:1035] T f77025fd71924964afafcd32c9ed6923 P 75bd3f0e8d7c4e8aa92a353301afeb21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:24.655088 26396 raft_consensus.cc:2932] T f77025fd71924964afafcd32c9ed6923 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 30921110704f47f88d68a7d9364d42cc (127.9.176.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } attrs { replace: true } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: true } } }
I20240502 01:55:24.655740 25133 raft_consensus.cc:2932] T f77025fd71924964afafcd32c9ed6923 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 30921110704f47f88d68a7d9364d42cc (127.9.176.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } attrs { replace: true } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: true } } }
I20240502 01:55:24.655902 24868 raft_consensus.cc:2932] T f77025fd71924964afafcd32c9ed6923 P 96e5c2fd5d624c2da735e72b387a12d9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 30921110704f47f88d68a7d9364d42cc (127.9.176.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } attrs { replace: true } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: true } } }
I20240502 01:55:24.660180 24722 catalog_manager.cc:5484] T f77025fd71924964afafcd32c9ed6923 P 672a55e7d4874c5d99f3c5e3d9550f15 reported cstate change: config changed from index -1 to 4, NON_VOTER 30921110704f47f88d68a7d9364d42cc (127.9.176.196) added. New cstate: current_term: 1 leader_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } attrs { replace: true } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: true } } }
I20240502 01:55:24.677021 25133 consensus_queue.cc:237] T b6beb3b47fd545d2b6a2d9181f4e5468 P 672a55e7d4874c5d99f3c5e3d9550f15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { replace: true } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: true } }
I20240502 01:55:24.679332 24868 raft_consensus.cc:1261] T b6beb3b47fd545d2b6a2d9181f4e5468 P 96e5c2fd5d624c2da735e72b387a12d9 [term 1 FOLLOWER]: Refusing update from remote peer 672a55e7d4874c5d99f3c5e3d9550f15: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:24.679595 25002 raft_consensus.cc:1261] T b6beb3b47fd545d2b6a2d9181f4e5468 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 1 FOLLOWER]: Refusing update from remote peer 672a55e7d4874c5d99f3c5e3d9550f15: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:24.680176 26410 consensus_queue.cc:1035] T b6beb3b47fd545d2b6a2d9181f4e5468 P 672a55e7d4874c5d99f3c5e3d9550f15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:24.680575 26413 consensus_queue.cc:1035] T b6beb3b47fd545d2b6a2d9181f4e5468 P 672a55e7d4874c5d99f3c5e3d9550f15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 01:55:24.680879 25067 consensus_peers.cc:485] T b6beb3b47fd545d2b6a2d9181f4e5468 P 672a55e7d4874c5d99f3c5e3d9550f15 -> Peer 6742b9db50c64ea394c2ca38ddee0872 (127.9.176.197:43479): Couldn't send request to peer 6742b9db50c64ea394c2ca38ddee0872. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b6beb3b47fd545d2b6a2d9181f4e5468. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:24.683655 26413 raft_consensus.cc:2932] T b6beb3b47fd545d2b6a2d9181f4e5468 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6742b9db50c64ea394c2ca38ddee0872 (127.9.176.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { replace: true } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: true } } }
I20240502 01:55:24.684458 24868 raft_consensus.cc:2932] T b6beb3b47fd545d2b6a2d9181f4e5468 P 96e5c2fd5d624c2da735e72b387a12d9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6742b9db50c64ea394c2ca38ddee0872 (127.9.176.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { replace: true } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: true } } }
I20240502 01:55:24.685137 25002 raft_consensus.cc:2932] T b6beb3b47fd545d2b6a2d9181f4e5468 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6742b9db50c64ea394c2ca38ddee0872 (127.9.176.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { replace: true } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: true } } }
I20240502 01:55:24.689463 24724 catalog_manager.cc:5484] T b6beb3b47fd545d2b6a2d9181f4e5468 P 96e5c2fd5d624c2da735e72b387a12d9 reported cstate change: config changed from index -1 to 4, NON_VOTER 6742b9db50c64ea394c2ca38ddee0872 (127.9.176.197) added. New cstate: current_term: 1 leader_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { replace: true } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: true } } }
I20240502 01:55:24.710811 25133 consensus_queue.cc:237] T a9c57322de524bce8a114d9a94bd019d P 672a55e7d4874c5d99f3c5e3d9550f15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } attrs { replace: true } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: true } }
I20240502 01:55:24.713656 25002 raft_consensus.cc:1261] T a9c57322de524bce8a114d9a94bd019d P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 1 FOLLOWER]: Refusing update from remote peer 672a55e7d4874c5d99f3c5e3d9550f15: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:24.713829 24868 raft_consensus.cc:1261] T a9c57322de524bce8a114d9a94bd019d P 96e5c2fd5d624c2da735e72b387a12d9 [term 1 FOLLOWER]: Refusing update from remote peer 672a55e7d4874c5d99f3c5e3d9550f15: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:24.714303 26439 consensus_queue.cc:1035] T a9c57322de524bce8a114d9a94bd019d P 672a55e7d4874c5d99f3c5e3d9550f15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:24.714697 26413 consensus_queue.cc:1035] T a9c57322de524bce8a114d9a94bd019d P 672a55e7d4874c5d99f3c5e3d9550f15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:24.717906 26413 raft_consensus.cc:2932] T a9c57322de524bce8a114d9a94bd019d P 672a55e7d4874c5d99f3c5e3d9550f15 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d490b54f0e954ef59deb6750523f6027 (127.9.176.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } attrs { replace: true } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: true } } }
I20240502 01:55:24.720402 25002 raft_consensus.cc:2932] T a9c57322de524bce8a114d9a94bd019d P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d490b54f0e954ef59deb6750523f6027 (127.9.176.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } attrs { replace: true } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: true } } }
W20240502 01:55:24.720808 25069 consensus_peers.cc:485] T a9c57322de524bce8a114d9a94bd019d P 672a55e7d4874c5d99f3c5e3d9550f15 -> Peer d490b54f0e954ef59deb6750523f6027 (127.9.176.198:34317): Couldn't send request to peer d490b54f0e954ef59deb6750523f6027. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a9c57322de524bce8a114d9a94bd019d. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:24.720867 24868 raft_consensus.cc:2932] T a9c57322de524bce8a114d9a94bd019d P 96e5c2fd5d624c2da735e72b387a12d9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d490b54f0e954ef59deb6750523f6027 (127.9.176.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } attrs { replace: true } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: true } } }
I20240502 01:55:24.726153 24724 catalog_manager.cc:5484] T a9c57322de524bce8a114d9a94bd019d P 672a55e7d4874c5d99f3c5e3d9550f15 reported cstate change: config changed from index -1 to 4, NON_VOTER d490b54f0e954ef59deb6750523f6027 (127.9.176.198) added. New cstate: current_term: 1 leader_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:55:24.740097 24868 consensus_queue.cc:237] T 32b0189cb19f42efbe7118003e9390f9 P 96e5c2fd5d624c2da735e72b387a12d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } attrs { replace: true } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: true } }
I20240502 01:55:24.743070 25002 raft_consensus.cc:1261] T 32b0189cb19f42efbe7118003e9390f9 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 1 FOLLOWER]: Refusing update from remote peer 96e5c2fd5d624c2da735e72b387a12d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:24.743768 25133 raft_consensus.cc:1261] T 32b0189cb19f42efbe7118003e9390f9 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 1 FOLLOWER]: Refusing update from remote peer 96e5c2fd5d624c2da735e72b387a12d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:24.744160 26454 consensus_queue.cc:1035] T 32b0189cb19f42efbe7118003e9390f9 P 96e5c2fd5d624c2da735e72b387a12d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:24.744552 26323 consensus_queue.cc:1035] T 32b0189cb19f42efbe7118003e9390f9 P 96e5c2fd5d624c2da735e72b387a12d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:24.747958 26455 raft_consensus.cc:2932] T 32b0189cb19f42efbe7118003e9390f9 P 96e5c2fd5d624c2da735e72b387a12d9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 30921110704f47f88d68a7d9364d42cc (127.9.176.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } attrs { replace: true } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: true } } }
I20240502 01:55:24.748692 25133 raft_consensus.cc:2932] T 32b0189cb19f42efbe7118003e9390f9 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 30921110704f47f88d68a7d9364d42cc (127.9.176.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } attrs { replace: true } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: true } } }
W20240502 01:55:24.753803 24805 consensus_peers.cc:485] T 32b0189cb19f42efbe7118003e9390f9 P 96e5c2fd5d624c2da735e72b387a12d9 -> Peer 30921110704f47f88d68a7d9364d42cc (127.9.176.196:39355): Couldn't send request to peer 30921110704f47f88d68a7d9364d42cc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 32b0189cb19f42efbe7118003e9390f9. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:24.754449 24725 catalog_manager.cc:5484] T 32b0189cb19f42efbe7118003e9390f9 P 672a55e7d4874c5d99f3c5e3d9550f15 reported cstate change: config changed from index -1 to 4, NON_VOTER 30921110704f47f88d68a7d9364d42cc (127.9.176.196) added. New cstate: current_term: 1 leader_uuid: "96e5c2fd5d624c2da735e72b387a12d9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } attrs { replace: true } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: true } } }
I20240502 01:55:24.754840 25002 raft_consensus.cc:2932] T 32b0189cb19f42efbe7118003e9390f9 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 30921110704f47f88d68a7d9364d42cc (127.9.176.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } attrs { replace: true } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: true } } }
I20240502 01:55:24.775077 24868 consensus_queue.cc:237] T 75d71cdcf4d44353acb7d2f18696977b P 96e5c2fd5d624c2da735e72b387a12d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { replace: true } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: true } }
I20240502 01:55:24.777760 25002 raft_consensus.cc:1261] T 75d71cdcf4d44353acb7d2f18696977b P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 1 FOLLOWER]: Refusing update from remote peer 96e5c2fd5d624c2da735e72b387a12d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:24.778054 25133 raft_consensus.cc:1261] T 75d71cdcf4d44353acb7d2f18696977b P 672a55e7d4874c5d99f3c5e3d9550f15 [term 1 FOLLOWER]: Refusing update from remote peer 96e5c2fd5d624c2da735e72b387a12d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:24.778601 26455 consensus_queue.cc:1035] T 75d71cdcf4d44353acb7d2f18696977b P 96e5c2fd5d624c2da735e72b387a12d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:24.778985 26459 consensus_queue.cc:1035] T 75d71cdcf4d44353acb7d2f18696977b P 96e5c2fd5d624c2da735e72b387a12d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:24.782444 26454 raft_consensus.cc:2932] T 75d71cdcf4d44353acb7d2f18696977b P 96e5c2fd5d624c2da735e72b387a12d9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6742b9db50c64ea394c2ca38ddee0872 (127.9.176.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { replace: true } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: true } } }
W20240502 01:55:24.785068 24802 consensus_peers.cc:485] T 75d71cdcf4d44353acb7d2f18696977b P 96e5c2fd5d624c2da735e72b387a12d9 -> Peer 6742b9db50c64ea394c2ca38ddee0872 (127.9.176.197:43479): Couldn't send request to peer 6742b9db50c64ea394c2ca38ddee0872. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 75d71cdcf4d44353acb7d2f18696977b. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:24.786509 25133 raft_consensus.cc:2932] T 75d71cdcf4d44353acb7d2f18696977b P 672a55e7d4874c5d99f3c5e3d9550f15 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6742b9db50c64ea394c2ca38ddee0872 (127.9.176.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { replace: true } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: true } } }
I20240502 01:55:24.786729 25002 raft_consensus.cc:2932] T 75d71cdcf4d44353acb7d2f18696977b P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6742b9db50c64ea394c2ca38ddee0872 (127.9.176.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { replace: true } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: true } } }
I20240502 01:55:24.791275 24724 catalog_manager.cc:5484] T 75d71cdcf4d44353acb7d2f18696977b P 96e5c2fd5d624c2da735e72b387a12d9 reported cstate change: config changed from index -1 to 4, NON_VOTER 6742b9db50c64ea394c2ca38ddee0872 (127.9.176.197) added. New cstate: current_term: 1 leader_uuid: "96e5c2fd5d624c2da735e72b387a12d9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:55:24.805747 25133 consensus_queue.cc:237] T ce37ee2e946b4075b78f3c9456939848 P 672a55e7d4874c5d99f3c5e3d9550f15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } attrs { replace: true } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: true } }
I20240502 01:55:24.808212 24868 raft_consensus.cc:1261] T ce37ee2e946b4075b78f3c9456939848 P 96e5c2fd5d624c2da735e72b387a12d9 [term 1 FOLLOWER]: Refusing update from remote peer 672a55e7d4874c5d99f3c5e3d9550f15: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 01:55:24.808477 25069 consensus_peers.cc:485] T ce37ee2e946b4075b78f3c9456939848 P 672a55e7d4874c5d99f3c5e3d9550f15 -> Peer d490b54f0e954ef59deb6750523f6027 (127.9.176.198:34317): Couldn't send request to peer d490b54f0e954ef59deb6750523f6027. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ce37ee2e946b4075b78f3c9456939848. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:24.808672 25002 raft_consensus.cc:1261] T ce37ee2e946b4075b78f3c9456939848 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 1 FOLLOWER]: Refusing update from remote peer 672a55e7d4874c5d99f3c5e3d9550f15: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:24.808838 26439 consensus_queue.cc:1035] T ce37ee2e946b4075b78f3c9456939848 P 672a55e7d4874c5d99f3c5e3d9550f15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:24.809280 26410 consensus_queue.cc:1035] T ce37ee2e946b4075b78f3c9456939848 P 672a55e7d4874c5d99f3c5e3d9550f15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:24.812124 26439 raft_consensus.cc:2932] T ce37ee2e946b4075b78f3c9456939848 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d490b54f0e954ef59deb6750523f6027 (127.9.176.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } attrs { replace: true } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: true } } }
I20240502 01:55:24.812978 24868 raft_consensus.cc:2932] T ce37ee2e946b4075b78f3c9456939848 P 96e5c2fd5d624c2da735e72b387a12d9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d490b54f0e954ef59deb6750523f6027 (127.9.176.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } attrs { replace: true } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: true } } }
I20240502 01:55:24.819151 24722 catalog_manager.cc:5484] T ce37ee2e946b4075b78f3c9456939848 P 672a55e7d4874c5d99f3c5e3d9550f15 reported cstate change: config changed from index -1 to 4, NON_VOTER d490b54f0e954ef59deb6750523f6027 (127.9.176.198) added. New cstate: current_term: 1 leader_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:55:24.819110 25002 raft_consensus.cc:2932] T ce37ee2e946b4075b78f3c9456939848 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d490b54f0e954ef59deb6750523f6027 (127.9.176.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } attrs { replace: true } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: true } } }
I20240502 01:55:25.057237 26522 ts_tablet_manager.cc:902] T 1b76f351e8644d9e8ff27e3f1a963ba2 P d490b54f0e954ef59deb6750523f6027: Initiating tablet copy from peer 75bd3f0e8d7c4e8aa92a353301afeb21 (127.9.176.194:41625)
I20240502 01:55:25.059140 26522 tablet_copy_client.cc:308] T 1b76f351e8644d9e8ff27e3f1a963ba2 P d490b54f0e954ef59deb6750523f6027: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.194:41625
I20240502 01:55:25.066174 25022 tablet_copy_service.cc:140] P 75bd3f0e8d7c4e8aa92a353301afeb21: Received BeginTabletCopySession request for tablet 1b76f351e8644d9e8ff27e3f1a963ba2 from peer d490b54f0e954ef59deb6750523f6027 ({username='slave'} at 127.9.176.198:39779)
I20240502 01:55:25.066498 25022 tablet_copy_service.cc:161] P 75bd3f0e8d7c4e8aa92a353301afeb21: Beginning new tablet copy session on tablet 1b76f351e8644d9e8ff27e3f1a963ba2 from peer d490b54f0e954ef59deb6750523f6027 at {username='slave'} at 127.9.176.198:39779: session id = d490b54f0e954ef59deb6750523f6027-1b76f351e8644d9e8ff27e3f1a963ba2
I20240502 01:55:25.069732 25022 tablet_copy_source_session.cc:205] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 75bd3f0e8d7c4e8aa92a353301afeb21: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:25.074302 26522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b76f351e8644d9e8ff27e3f1a963ba2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:25.085130 26522 tablet_copy_client.cc:783] T 1b76f351e8644d9e8ff27e3f1a963ba2 P d490b54f0e954ef59deb6750523f6027: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:25.085525 26522 tablet_copy_client.cc:647] T 1b76f351e8644d9e8ff27e3f1a963ba2 P d490b54f0e954ef59deb6750523f6027: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:25.088142 26522 tablet_copy_client.cc:523] T 1b76f351e8644d9e8ff27e3f1a963ba2 P d490b54f0e954ef59deb6750523f6027: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:25.091450 26522 tablet_bootstrap.cc:492] T 1b76f351e8644d9e8ff27e3f1a963ba2 P d490b54f0e954ef59deb6750523f6027: Bootstrap starting.
I20240502 01:55:25.101948 26522 log.cc:824] T 1b76f351e8644d9e8ff27e3f1a963ba2 P d490b54f0e954ef59deb6750523f6027: Log is configured to *not* fsync() on all Append() calls
I20240502 01:55:25.111305 26529 ts_tablet_manager.cc:902] T f77025fd71924964afafcd32c9ed6923 P 30921110704f47f88d68a7d9364d42cc: Initiating tablet copy from peer 75bd3f0e8d7c4e8aa92a353301afeb21 (127.9.176.194:41625)
I20240502 01:55:25.113484 26529 tablet_copy_client.cc:308] T f77025fd71924964afafcd32c9ed6923 P 30921110704f47f88d68a7d9364d42cc: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.194:41625
I20240502 01:55:25.122571 25022 tablet_copy_service.cc:140] P 75bd3f0e8d7c4e8aa92a353301afeb21: Received BeginTabletCopySession request for tablet f77025fd71924964afafcd32c9ed6923 from peer 30921110704f47f88d68a7d9364d42cc ({username='slave'} at 127.9.176.196:56167)
I20240502 01:55:25.122794 25022 tablet_copy_service.cc:161] P 75bd3f0e8d7c4e8aa92a353301afeb21: Beginning new tablet copy session on tablet f77025fd71924964afafcd32c9ed6923 from peer 30921110704f47f88d68a7d9364d42cc at {username='slave'} at 127.9.176.196:56167: session id = 30921110704f47f88d68a7d9364d42cc-f77025fd71924964afafcd32c9ed6923
I20240502 01:55:25.125586 25022 tablet_copy_source_session.cc:205] T f77025fd71924964afafcd32c9ed6923 P 75bd3f0e8d7c4e8aa92a353301afeb21: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:25.130167 26529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f77025fd71924964afafcd32c9ed6923. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:25.143752 26529 tablet_copy_client.cc:783] T f77025fd71924964afafcd32c9ed6923 P 30921110704f47f88d68a7d9364d42cc: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:25.144157 26529 tablet_copy_client.cc:647] T f77025fd71924964afafcd32c9ed6923 P 30921110704f47f88d68a7d9364d42cc: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:25.146507 26529 tablet_copy_client.cc:523] T f77025fd71924964afafcd32c9ed6923 P 30921110704f47f88d68a7d9364d42cc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:25.148267 26522 tablet_bootstrap.cc:492] T 1b76f351e8644d9e8ff27e3f1a963ba2 P d490b54f0e954ef59deb6750523f6027: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:25.148927 26522 tablet_bootstrap.cc:492] T 1b76f351e8644d9e8ff27e3f1a963ba2 P d490b54f0e954ef59deb6750523f6027: Bootstrap complete.
I20240502 01:55:25.149500 26522 ts_tablet_manager.cc:1367] T 1b76f351e8644d9e8ff27e3f1a963ba2 P d490b54f0e954ef59deb6750523f6027: Time spent bootstrapping tablet: real 0.058s	user 0.056s	sys 0.004s
I20240502 01:55:25.151270 26529 tablet_bootstrap.cc:492] T f77025fd71924964afafcd32c9ed6923 P 30921110704f47f88d68a7d9364d42cc: Bootstrap starting.
I20240502 01:55:25.155803 26522 raft_consensus.cc:348] T 1b76f351e8644d9e8ff27e3f1a963ba2 P d490b54f0e954ef59deb6750523f6027 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } attrs { replace: true } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: true } }
I20240502 01:55:25.157306 26522 raft_consensus.cc:729] T 1b76f351e8644d9e8ff27e3f1a963ba2 P d490b54f0e954ef59deb6750523f6027 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d490b54f0e954ef59deb6750523f6027, State: Initialized, Role: LEARNER
I20240502 01:55:25.158140 26522 consensus_queue.cc:260] T 1b76f351e8644d9e8ff27e3f1a963ba2 P d490b54f0e954ef59deb6750523f6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } attrs { replace: true } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: true } }
I20240502 01:55:25.160651 26522 ts_tablet_manager.cc:1398] T 1b76f351e8644d9e8ff27e3f1a963ba2 P d490b54f0e954ef59deb6750523f6027: Time spent starting tablet: real 0.011s	user 0.009s	sys 0.000s
I20240502 01:55:25.161758 26550 ts_tablet_manager.cc:902] T a9c57322de524bce8a114d9a94bd019d P d490b54f0e954ef59deb6750523f6027: Initiating tablet copy from peer 672a55e7d4874c5d99f3c5e3d9550f15 (127.9.176.195:46579)
I20240502 01:55:25.161841 25022 tablet_copy_service.cc:342] P 75bd3f0e8d7c4e8aa92a353301afeb21: Request end of tablet copy session d490b54f0e954ef59deb6750523f6027-1b76f351e8644d9e8ff27e3f1a963ba2 received from {username='slave'} at 127.9.176.198:39779
I20240502 01:55:25.162035 25022 tablet_copy_service.cc:433] P 75bd3f0e8d7c4e8aa92a353301afeb21: ending tablet copy session d490b54f0e954ef59deb6750523f6027-1b76f351e8644d9e8ff27e3f1a963ba2 on tablet 1b76f351e8644d9e8ff27e3f1a963ba2 with peer d490b54f0e954ef59deb6750523f6027
I20240502 01:55:25.165643 26529 log.cc:824] T f77025fd71924964afafcd32c9ed6923 P 30921110704f47f88d68a7d9364d42cc: Log is configured to *not* fsync() on all Append() calls
I20240502 01:55:25.200042 26522 ts_tablet_manager.cc:902] T ce37ee2e946b4075b78f3c9456939848 P d490b54f0e954ef59deb6750523f6027: Initiating tablet copy from peer 672a55e7d4874c5d99f3c5e3d9550f15 (127.9.176.195:46579)
I20240502 01:55:25.217058 26560 ts_tablet_manager.cc:902] T 450538b8217d40ce945d51e2c171ee36 P 6742b9db50c64ea394c2ca38ddee0872: Initiating tablet copy from peer 672a55e7d4874c5d99f3c5e3d9550f15 (127.9.176.195:46579)
I20240502 01:55:25.217414 26550 tablet_copy_client.cc:308] T a9c57322de524bce8a114d9a94bd019d P d490b54f0e954ef59deb6750523f6027: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.195:46579
I20240502 01:55:25.221921 26522 tablet_copy_client.cc:308] T ce37ee2e946b4075b78f3c9456939848 P d490b54f0e954ef59deb6750523f6027: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.195:46579
I20240502 01:55:25.228286 26556 ts_tablet_manager.cc:902] T 08bb5eb2c2814019b6eda6fbbc288245 P 30921110704f47f88d68a7d9364d42cc: Initiating tablet copy from peer 75bd3f0e8d7c4e8aa92a353301afeb21 (127.9.176.194:41625)
I20240502 01:55:25.245128 25154 tablet_copy_service.cc:140] P 672a55e7d4874c5d99f3c5e3d9550f15: Received BeginTabletCopySession request for tablet a9c57322de524bce8a114d9a94bd019d from peer d490b54f0e954ef59deb6750523f6027 ({username='slave'} at 127.9.176.198:40081)
I20240502 01:55:25.245462 25154 tablet_copy_service.cc:161] P 672a55e7d4874c5d99f3c5e3d9550f15: Beginning new tablet copy session on tablet a9c57322de524bce8a114d9a94bd019d from peer d490b54f0e954ef59deb6750523f6027 at {username='slave'} at 127.9.176.198:40081: session id = d490b54f0e954ef59deb6750523f6027-a9c57322de524bce8a114d9a94bd019d
I20240502 01:55:25.250990 25154 tablet_copy_source_session.cc:205] T a9c57322de524bce8a114d9a94bd019d P 672a55e7d4874c5d99f3c5e3d9550f15: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:25.251833 25153 tablet_copy_service.cc:140] P 672a55e7d4874c5d99f3c5e3d9550f15: Received BeginTabletCopySession request for tablet ce37ee2e946b4075b78f3c9456939848 from peer d490b54f0e954ef59deb6750523f6027 ({username='slave'} at 127.9.176.198:40081)
I20240502 01:55:25.252178 25153 tablet_copy_service.cc:161] P 672a55e7d4874c5d99f3c5e3d9550f15: Beginning new tablet copy session on tablet ce37ee2e946b4075b78f3c9456939848 from peer d490b54f0e954ef59deb6750523f6027 at {username='slave'} at 127.9.176.198:40081: session id = d490b54f0e954ef59deb6750523f6027-ce37ee2e946b4075b78f3c9456939848
I20240502 01:55:25.256867 25153 tablet_copy_source_session.cc:205] T ce37ee2e946b4075b78f3c9456939848 P 672a55e7d4874c5d99f3c5e3d9550f15: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:25.259673 26522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce37ee2e946b4075b78f3c9456939848. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:25.264367 26529 tablet_bootstrap.cc:492] T f77025fd71924964afafcd32c9ed6923 P 30921110704f47f88d68a7d9364d42cc: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:25.265153 26529 tablet_bootstrap.cc:492] T f77025fd71924964afafcd32c9ed6923 P 30921110704f47f88d68a7d9364d42cc: Bootstrap complete.
I20240502 01:55:25.265799 26529 ts_tablet_manager.cc:1367] T f77025fd71924964afafcd32c9ed6923 P 30921110704f47f88d68a7d9364d42cc: Time spent bootstrapping tablet: real 0.115s	user 0.075s	sys 0.007s
I20240502 01:55:25.269431 26550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9c57322de524bce8a114d9a94bd019d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:25.274442 26529 raft_consensus.cc:348] T f77025fd71924964afafcd32c9ed6923 P 30921110704f47f88d68a7d9364d42cc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } attrs { replace: true } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: true } }
I20240502 01:55:25.275130 26529 raft_consensus.cc:729] T f77025fd71924964afafcd32c9ed6923 P 30921110704f47f88d68a7d9364d42cc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 30921110704f47f88d68a7d9364d42cc, State: Initialized, Role: LEARNER
I20240502 01:55:25.275936 26529 consensus_queue.cc:260] T f77025fd71924964afafcd32c9ed6923 P 30921110704f47f88d68a7d9364d42cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } attrs { replace: true } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: true } }
I20240502 01:55:25.277456 26575 ts_tablet_manager.cc:902] T b6beb3b47fd545d2b6a2d9181f4e5468 P 6742b9db50c64ea394c2ca38ddee0872: Initiating tablet copy from peer 672a55e7d4874c5d99f3c5e3d9550f15 (127.9.176.195:46579)
I20240502 01:55:25.279446 26550 tablet_copy_client.cc:783] T a9c57322de524bce8a114d9a94bd019d P d490b54f0e954ef59deb6750523f6027: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:25.279768 26550 tablet_copy_client.cc:647] T a9c57322de524bce8a114d9a94bd019d P d490b54f0e954ef59deb6750523f6027: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:25.280656 26522 tablet_copy_client.cc:783] T ce37ee2e946b4075b78f3c9456939848 P d490b54f0e954ef59deb6750523f6027: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:25.280926 26522 tablet_copy_client.cc:647] T ce37ee2e946b4075b78f3c9456939848 P d490b54f0e954ef59deb6750523f6027: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:25.284996 26522 tablet_copy_client.cc:523] T ce37ee2e946b4075b78f3c9456939848 P d490b54f0e954ef59deb6750523f6027: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:25.287122 26560 tablet_copy_client.cc:308] T 450538b8217d40ce945d51e2c171ee36 P 6742b9db50c64ea394c2ca38ddee0872: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.195:46579
I20240502 01:55:25.288869 26550 tablet_copy_client.cc:523] T a9c57322de524bce8a114d9a94bd019d P d490b54f0e954ef59deb6750523f6027: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:25.292548 26529 ts_tablet_manager.cc:1398] T f77025fd71924964afafcd32c9ed6923 P 30921110704f47f88d68a7d9364d42cc: Time spent starting tablet: real 0.027s	user 0.016s	sys 0.007s
I20240502 01:55:25.293790 25022 tablet_copy_service.cc:342] P 75bd3f0e8d7c4e8aa92a353301afeb21: Request end of tablet copy session 30921110704f47f88d68a7d9364d42cc-f77025fd71924964afafcd32c9ed6923 received from {username='slave'} at 127.9.176.196:56167
I20240502 01:55:25.293942 25022 tablet_copy_service.cc:433] P 75bd3f0e8d7c4e8aa92a353301afeb21: ending tablet copy session 30921110704f47f88d68a7d9364d42cc-f77025fd71924964afafcd32c9ed6923 on tablet f77025fd71924964afafcd32c9ed6923 with peer 30921110704f47f88d68a7d9364d42cc
I20240502 01:55:25.293974 26522 tablet_bootstrap.cc:492] T ce37ee2e946b4075b78f3c9456939848 P d490b54f0e954ef59deb6750523f6027: Bootstrap starting.
I20240502 01:55:25.295123 26550 tablet_bootstrap.cc:492] T a9c57322de524bce8a114d9a94bd019d P d490b54f0e954ef59deb6750523f6027: Bootstrap starting.
I20240502 01:55:25.314685 26586 ts_tablet_manager.cc:902] T 75d71cdcf4d44353acb7d2f18696977b P 6742b9db50c64ea394c2ca38ddee0872: Initiating tablet copy from peer 96e5c2fd5d624c2da735e72b387a12d9 (127.9.176.193:32783)
I20240502 01:55:25.322170 26575 tablet_copy_client.cc:308] T b6beb3b47fd545d2b6a2d9181f4e5468 P 6742b9db50c64ea394c2ca38ddee0872: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.195:46579
I20240502 01:55:25.323238 26556 tablet_copy_client.cc:308] T 08bb5eb2c2814019b6eda6fbbc288245 P 30921110704f47f88d68a7d9364d42cc: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.194:41625
I20240502 01:55:25.324426 25022 tablet_copy_service.cc:140] P 75bd3f0e8d7c4e8aa92a353301afeb21: Received BeginTabletCopySession request for tablet 08bb5eb2c2814019b6eda6fbbc288245 from peer 30921110704f47f88d68a7d9364d42cc ({username='slave'} at 127.9.176.196:56167)
I20240502 01:55:25.324671 25022 tablet_copy_service.cc:161] P 75bd3f0e8d7c4e8aa92a353301afeb21: Beginning new tablet copy session on tablet 08bb5eb2c2814019b6eda6fbbc288245 from peer 30921110704f47f88d68a7d9364d42cc at {username='slave'} at 127.9.176.196:56167: session id = 30921110704f47f88d68a7d9364d42cc-08bb5eb2c2814019b6eda6fbbc288245
I20240502 01:55:25.327894 26586 tablet_copy_client.cc:308] T 75d71cdcf4d44353acb7d2f18696977b P 6742b9db50c64ea394c2ca38ddee0872: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.193:32783
I20240502 01:55:25.328831 25022 tablet_copy_source_session.cc:205] T 08bb5eb2c2814019b6eda6fbbc288245 P 75bd3f0e8d7c4e8aa92a353301afeb21: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:25.329998 25154 tablet_copy_service.cc:140] P 672a55e7d4874c5d99f3c5e3d9550f15: Received BeginTabletCopySession request for tablet 450538b8217d40ce945d51e2c171ee36 from peer 6742b9db50c64ea394c2ca38ddee0872 ({username='slave'} at 127.9.176.197:38695)
I20240502 01:55:25.330224 25154 tablet_copy_service.cc:161] P 672a55e7d4874c5d99f3c5e3d9550f15: Beginning new tablet copy session on tablet 450538b8217d40ce945d51e2c171ee36 from peer 6742b9db50c64ea394c2ca38ddee0872 at {username='slave'} at 127.9.176.197:38695: session id = 6742b9db50c64ea394c2ca38ddee0872-450538b8217d40ce945d51e2c171ee36
I20240502 01:55:25.331063 26556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08bb5eb2c2814019b6eda6fbbc288245. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:25.333611 25154 tablet_copy_source_session.cc:205] T 450538b8217d40ce945d51e2c171ee36 P 672a55e7d4874c5d99f3c5e3d9550f15: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:25.334196 25153 tablet_copy_service.cc:140] P 672a55e7d4874c5d99f3c5e3d9550f15: Received BeginTabletCopySession request for tablet b6beb3b47fd545d2b6a2d9181f4e5468 from peer 6742b9db50c64ea394c2ca38ddee0872 ({username='slave'} at 127.9.176.197:38695)
I20240502 01:55:25.334367 25153 tablet_copy_service.cc:161] P 672a55e7d4874c5d99f3c5e3d9550f15: Beginning new tablet copy session on tablet b6beb3b47fd545d2b6a2d9181f4e5468 from peer 6742b9db50c64ea394c2ca38ddee0872 at {username='slave'} at 127.9.176.197:38695: session id = 6742b9db50c64ea394c2ca38ddee0872-b6beb3b47fd545d2b6a2d9181f4e5468
I20240502 01:55:25.337823 25153 tablet_copy_source_session.cc:205] T b6beb3b47fd545d2b6a2d9181f4e5468 P 672a55e7d4874c5d99f3c5e3d9550f15: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:25.340660 24889 tablet_copy_service.cc:140] P 96e5c2fd5d624c2da735e72b387a12d9: Received BeginTabletCopySession request for tablet 75d71cdcf4d44353acb7d2f18696977b from peer 6742b9db50c64ea394c2ca38ddee0872 ({username='slave'} at 127.9.176.197:39001)
I20240502 01:55:25.340895 24889 tablet_copy_service.cc:161] P 96e5c2fd5d624c2da735e72b387a12d9: Beginning new tablet copy session on tablet 75d71cdcf4d44353acb7d2f18696977b from peer 6742b9db50c64ea394c2ca38ddee0872 at {username='slave'} at 127.9.176.197:39001: session id = 6742b9db50c64ea394c2ca38ddee0872-75d71cdcf4d44353acb7d2f18696977b
I20240502 01:55:25.343353 26560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 450538b8217d40ce945d51e2c171ee36. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:25.344305 24889 tablet_copy_source_session.cc:205] T 75d71cdcf4d44353acb7d2f18696977b P 96e5c2fd5d624c2da735e72b387a12d9: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:25.345264 26556 tablet_copy_client.cc:783] T 08bb5eb2c2814019b6eda6fbbc288245 P 30921110704f47f88d68a7d9364d42cc: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:25.345824 26556 tablet_copy_client.cc:647] T 08bb5eb2c2814019b6eda6fbbc288245 P 30921110704f47f88d68a7d9364d42cc: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:25.347754 26575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6beb3b47fd545d2b6a2d9181f4e5468. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:25.348892 26556 tablet_copy_client.cc:523] T 08bb5eb2c2814019b6eda6fbbc288245 P 30921110704f47f88d68a7d9364d42cc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:25.348997 26586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75d71cdcf4d44353acb7d2f18696977b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:25.354246 26556 tablet_bootstrap.cc:492] T 08bb5eb2c2814019b6eda6fbbc288245 P 30921110704f47f88d68a7d9364d42cc: Bootstrap starting.
I20240502 01:55:25.356431 26529 ts_tablet_manager.cc:902] T 32b0189cb19f42efbe7118003e9390f9 P 30921110704f47f88d68a7d9364d42cc: Initiating tablet copy from peer 96e5c2fd5d624c2da735e72b387a12d9 (127.9.176.193:32783)
I20240502 01:55:25.366703 26529 tablet_copy_client.cc:308] T 32b0189cb19f42efbe7118003e9390f9 P 30921110704f47f88d68a7d9364d42cc: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.193:32783
I20240502 01:55:25.375974 24889 tablet_copy_service.cc:140] P 96e5c2fd5d624c2da735e72b387a12d9: Received BeginTabletCopySession request for tablet 32b0189cb19f42efbe7118003e9390f9 from peer 30921110704f47f88d68a7d9364d42cc ({username='slave'} at 127.9.176.196:36527)
I20240502 01:55:25.376200 24889 tablet_copy_service.cc:161] P 96e5c2fd5d624c2da735e72b387a12d9: Beginning new tablet copy session on tablet 32b0189cb19f42efbe7118003e9390f9 from peer 30921110704f47f88d68a7d9364d42cc at {username='slave'} at 127.9.176.196:36527: session id = 30921110704f47f88d68a7d9364d42cc-32b0189cb19f42efbe7118003e9390f9
I20240502 01:55:25.379321 24889 tablet_copy_source_session.cc:205] T 32b0189cb19f42efbe7118003e9390f9 P 96e5c2fd5d624c2da735e72b387a12d9: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:25.380980 26586 tablet_copy_client.cc:783] T 75d71cdcf4d44353acb7d2f18696977b P 6742b9db50c64ea394c2ca38ddee0872: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:25.381201 26529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32b0189cb19f42efbe7118003e9390f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:25.381430 26586 tablet_copy_client.cc:647] T 75d71cdcf4d44353acb7d2f18696977b P 6742b9db50c64ea394c2ca38ddee0872: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:25.383476 26575 tablet_copy_client.cc:783] T b6beb3b47fd545d2b6a2d9181f4e5468 P 6742b9db50c64ea394c2ca38ddee0872: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:25.383754 26575 tablet_copy_client.cc:647] T b6beb3b47fd545d2b6a2d9181f4e5468 P 6742b9db50c64ea394c2ca38ddee0872: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:25.385918 26575 tablet_copy_client.cc:523] T b6beb3b47fd545d2b6a2d9181f4e5468 P 6742b9db50c64ea394c2ca38ddee0872: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:25.390435 26586 tablet_copy_client.cc:523] T 75d71cdcf4d44353acb7d2f18696977b P 6742b9db50c64ea394c2ca38ddee0872: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:25.390651 26560 tablet_copy_client.cc:783] T 450538b8217d40ce945d51e2c171ee36 P 6742b9db50c64ea394c2ca38ddee0872: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:25.390944 26560 tablet_copy_client.cc:647] T 450538b8217d40ce945d51e2c171ee36 P 6742b9db50c64ea394c2ca38ddee0872: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:25.393046 26560 tablet_copy_client.cc:523] T 450538b8217d40ce945d51e2c171ee36 P 6742b9db50c64ea394c2ca38ddee0872: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:25.395854 26586 tablet_bootstrap.cc:492] T 75d71cdcf4d44353acb7d2f18696977b P 6742b9db50c64ea394c2ca38ddee0872: Bootstrap starting.
I20240502 01:55:25.397667 26560 tablet_bootstrap.cc:492] T 450538b8217d40ce945d51e2c171ee36 P 6742b9db50c64ea394c2ca38ddee0872: Bootstrap starting.
I20240502 01:55:25.399682 26529 tablet_copy_client.cc:783] T 32b0189cb19f42efbe7118003e9390f9 P 30921110704f47f88d68a7d9364d42cc: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:25.400158 26529 tablet_copy_client.cc:647] T 32b0189cb19f42efbe7118003e9390f9 P 30921110704f47f88d68a7d9364d42cc: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:25.398329 26575 tablet_bootstrap.cc:492] T b6beb3b47fd545d2b6a2d9181f4e5468 P 6742b9db50c64ea394c2ca38ddee0872: Bootstrap starting.
I20240502 01:55:25.402570 26529 tablet_copy_client.cc:523] T 32b0189cb19f42efbe7118003e9390f9 P 30921110704f47f88d68a7d9364d42cc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:25.416546 26529 tablet_bootstrap.cc:492] T 32b0189cb19f42efbe7118003e9390f9 P 30921110704f47f88d68a7d9364d42cc: Bootstrap starting.
I20240502 01:55:25.418933 26575 log.cc:824] T b6beb3b47fd545d2b6a2d9181f4e5468 P 6742b9db50c64ea394c2ca38ddee0872: Log is configured to *not* fsync() on all Append() calls
I20240502 01:55:25.447775 26550 tablet_bootstrap.cc:492] T a9c57322de524bce8a114d9a94bd019d P d490b54f0e954ef59deb6750523f6027: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:25.448359 26550 tablet_bootstrap.cc:492] T a9c57322de524bce8a114d9a94bd019d P d490b54f0e954ef59deb6750523f6027: Bootstrap complete.
I20240502 01:55:25.448729 26550 ts_tablet_manager.cc:1367] T a9c57322de524bce8a114d9a94bd019d P d490b54f0e954ef59deb6750523f6027: Time spent bootstrapping tablet: real 0.154s	user 0.067s	sys 0.038s
I20240502 01:55:25.449806 26550 raft_consensus.cc:348] T a9c57322de524bce8a114d9a94bd019d P d490b54f0e954ef59deb6750523f6027 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } attrs { replace: true } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: true } }
I20240502 01:55:25.450191 26550 raft_consensus.cc:729] T a9c57322de524bce8a114d9a94bd019d P d490b54f0e954ef59deb6750523f6027 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d490b54f0e954ef59deb6750523f6027, State: Initialized, Role: LEARNER
I20240502 01:55:25.450474 26550 consensus_queue.cc:260] T a9c57322de524bce8a114d9a94bd019d P d490b54f0e954ef59deb6750523f6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } attrs { replace: true } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: true } }
I20240502 01:55:25.454200 26550 ts_tablet_manager.cc:1398] T a9c57322de524bce8a114d9a94bd019d P d490b54f0e954ef59deb6750523f6027: Time spent starting tablet: real 0.005s	user 0.002s	sys 0.000s
I20240502 01:55:25.457661 25154 tablet_copy_service.cc:342] P 672a55e7d4874c5d99f3c5e3d9550f15: Request end of tablet copy session d490b54f0e954ef59deb6750523f6027-a9c57322de524bce8a114d9a94bd019d received from {username='slave'} at 127.9.176.198:40081
I20240502 01:55:25.457850 25154 tablet_copy_service.cc:433] P 672a55e7d4874c5d99f3c5e3d9550f15: ending tablet copy session d490b54f0e954ef59deb6750523f6027-a9c57322de524bce8a114d9a94bd019d on tablet a9c57322de524bce8a114d9a94bd019d with peer d490b54f0e954ef59deb6750523f6027
I20240502 01:55:25.495421 26522 tablet_bootstrap.cc:492] T ce37ee2e946b4075b78f3c9456939848 P d490b54f0e954ef59deb6750523f6027: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:25.499745 26575 tablet_bootstrap.cc:492] T b6beb3b47fd545d2b6a2d9181f4e5468 P 6742b9db50c64ea394c2ca38ddee0872: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:25.500483 26575 tablet_bootstrap.cc:492] T b6beb3b47fd545d2b6a2d9181f4e5468 P 6742b9db50c64ea394c2ca38ddee0872: Bootstrap complete.
I20240502 01:55:25.501091 26575 ts_tablet_manager.cc:1367] T b6beb3b47fd545d2b6a2d9181f4e5468 P 6742b9db50c64ea394c2ca38ddee0872: Time spent bootstrapping tablet: real 0.103s	user 0.055s	sys 0.008s
I20240502 01:55:25.509387 26522 tablet_bootstrap.cc:492] T ce37ee2e946b4075b78f3c9456939848 P d490b54f0e954ef59deb6750523f6027: Bootstrap complete.
I20240502 01:55:25.509878 26522 ts_tablet_manager.cc:1367] T ce37ee2e946b4075b78f3c9456939848 P d490b54f0e954ef59deb6750523f6027: Time spent bootstrapping tablet: real 0.216s	user 0.067s	sys 0.008s
I20240502 01:55:25.512090 26522 raft_consensus.cc:348] T ce37ee2e946b4075b78f3c9456939848 P d490b54f0e954ef59deb6750523f6027 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } attrs { replace: true } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: true } }
I20240502 01:55:25.513177 26522 raft_consensus.cc:729] T ce37ee2e946b4075b78f3c9456939848 P d490b54f0e954ef59deb6750523f6027 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d490b54f0e954ef59deb6750523f6027, State: Initialized, Role: LEARNER
I20240502 01:55:25.513468 26522 consensus_queue.cc:260] T ce37ee2e946b4075b78f3c9456939848 P d490b54f0e954ef59deb6750523f6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } attrs { replace: true } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: true } }
I20240502 01:55:25.517546 26522 ts_tablet_manager.cc:1398] T ce37ee2e946b4075b78f3c9456939848 P d490b54f0e954ef59deb6750523f6027: Time spent starting tablet: real 0.006s	user 0.002s	sys 0.000s
I20240502 01:55:25.518682 26586 tablet_bootstrap.cc:492] T 75d71cdcf4d44353acb7d2f18696977b P 6742b9db50c64ea394c2ca38ddee0872: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:25.519178 26586 tablet_bootstrap.cc:492] T 75d71cdcf4d44353acb7d2f18696977b P 6742b9db50c64ea394c2ca38ddee0872: Bootstrap complete.
I20240502 01:55:25.519477 26586 ts_tablet_manager.cc:1367] T 75d71cdcf4d44353acb7d2f18696977b P 6742b9db50c64ea394c2ca38ddee0872: Time spent bootstrapping tablet: real 0.124s	user 0.043s	sys 0.017s
I20240502 01:55:25.508373 26575 raft_consensus.cc:348] T b6beb3b47fd545d2b6a2d9181f4e5468 P 6742b9db50c64ea394c2ca38ddee0872 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { replace: true } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: true } }
I20240502 01:55:25.519832 26575 raft_consensus.cc:729] T b6beb3b47fd545d2b6a2d9181f4e5468 P 6742b9db50c64ea394c2ca38ddee0872 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6742b9db50c64ea394c2ca38ddee0872, State: Initialized, Role: LEARNER
I20240502 01:55:25.520656 26586 raft_consensus.cc:348] T 75d71cdcf4d44353acb7d2f18696977b P 6742b9db50c64ea394c2ca38ddee0872 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { replace: true } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: true } }
I20240502 01:55:25.520556 26575 consensus_queue.cc:260] T b6beb3b47fd545d2b6a2d9181f4e5468 P 6742b9db50c64ea394c2ca38ddee0872 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { replace: true } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: true } }
I20240502 01:55:25.520889 26586 raft_consensus.cc:729] T 75d71cdcf4d44353acb7d2f18696977b P 6742b9db50c64ea394c2ca38ddee0872 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6742b9db50c64ea394c2ca38ddee0872, State: Initialized, Role: LEARNER
I20240502 01:55:25.521148 26586 consensus_queue.cc:260] T 75d71cdcf4d44353acb7d2f18696977b P 6742b9db50c64ea394c2ca38ddee0872 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { replace: true } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: true } }
I20240502 01:55:25.523907 26066 raft_consensus.cc:1203] T 1b76f351e8644d9e8ff27e3f1a963ba2 P d490b54f0e954ef59deb6750523f6027 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 01:55:25.535676 26560 tablet_bootstrap.cc:492] T 450538b8217d40ce945d51e2c171ee36 P 6742b9db50c64ea394c2ca38ddee0872: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:25.536306 26560 tablet_bootstrap.cc:492] T 450538b8217d40ce945d51e2c171ee36 P 6742b9db50c64ea394c2ca38ddee0872: Bootstrap complete.
I20240502 01:55:25.536710 26560 ts_tablet_manager.cc:1367] T 450538b8217d40ce945d51e2c171ee36 P 6742b9db50c64ea394c2ca38ddee0872: Time spent bootstrapping tablet: real 0.139s	user 0.059s	sys 0.007s
I20240502 01:55:25.537822 26560 raft_consensus.cc:348] T 450538b8217d40ce945d51e2c171ee36 P 6742b9db50c64ea394c2ca38ddee0872 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { replace: true } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: true } }
I20240502 01:55:25.538210 26560 raft_consensus.cc:729] T 450538b8217d40ce945d51e2c171ee36 P 6742b9db50c64ea394c2ca38ddee0872 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6742b9db50c64ea394c2ca38ddee0872, State: Initialized, Role: LEARNER
I20240502 01:55:25.538470 26560 consensus_queue.cc:260] T 450538b8217d40ce945d51e2c171ee36 P 6742b9db50c64ea394c2ca38ddee0872 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { replace: true } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: true } }
I20240502 01:55:25.544855 25154 tablet_copy_service.cc:342] P 672a55e7d4874c5d99f3c5e3d9550f15: Request end of tablet copy session d490b54f0e954ef59deb6750523f6027-ce37ee2e946b4075b78f3c9456939848 received from {username='slave'} at 127.9.176.198:40081
I20240502 01:55:25.545071 25154 tablet_copy_service.cc:433] P 672a55e7d4874c5d99f3c5e3d9550f15: ending tablet copy session d490b54f0e954ef59deb6750523f6027-ce37ee2e946b4075b78f3c9456939848 on tablet ce37ee2e946b4075b78f3c9456939848 with peer d490b54f0e954ef59deb6750523f6027
I20240502 01:55:25.553012 26586 ts_tablet_manager.cc:1398] T 75d71cdcf4d44353acb7d2f18696977b P 6742b9db50c64ea394c2ca38ddee0872: Time spent starting tablet: real 0.033s	user 0.006s	sys 0.013s
I20240502 01:55:25.553968 26575 ts_tablet_manager.cc:1398] T b6beb3b47fd545d2b6a2d9181f4e5468 P 6742b9db50c64ea394c2ca38ddee0872: Time spent starting tablet: real 0.053s	user 0.014s	sys 0.011s
I20240502 01:55:25.554359 24889 tablet_copy_service.cc:342] P 96e5c2fd5d624c2da735e72b387a12d9: Request end of tablet copy session 6742b9db50c64ea394c2ca38ddee0872-75d71cdcf4d44353acb7d2f18696977b received from {username='slave'} at 127.9.176.197:39001
I20240502 01:55:25.554574 24889 tablet_copy_service.cc:433] P 96e5c2fd5d624c2da735e72b387a12d9: ending tablet copy session 6742b9db50c64ea394c2ca38ddee0872-75d71cdcf4d44353acb7d2f18696977b on tablet 75d71cdcf4d44353acb7d2f18696977b with peer 6742b9db50c64ea394c2ca38ddee0872
I20240502 01:55:25.556474 25154 tablet_copy_service.cc:342] P 672a55e7d4874c5d99f3c5e3d9550f15: Request end of tablet copy session 6742b9db50c64ea394c2ca38ddee0872-b6beb3b47fd545d2b6a2d9181f4e5468 received from {username='slave'} at 127.9.176.197:38695
I20240502 01:55:25.556660 25154 tablet_copy_service.cc:433] P 672a55e7d4874c5d99f3c5e3d9550f15: ending tablet copy session 6742b9db50c64ea394c2ca38ddee0872-b6beb3b47fd545d2b6a2d9181f4e5468 on tablet b6beb3b47fd545d2b6a2d9181f4e5468 with peer 6742b9db50c64ea394c2ca38ddee0872
I20240502 01:55:25.571525 26560 ts_tablet_manager.cc:1398] T 450538b8217d40ce945d51e2c171ee36 P 6742b9db50c64ea394c2ca38ddee0872: Time spent starting tablet: real 0.035s	user 0.016s	sys 0.016s
I20240502 01:55:25.572609 25154 tablet_copy_service.cc:342] P 672a55e7d4874c5d99f3c5e3d9550f15: Request end of tablet copy session 6742b9db50c64ea394c2ca38ddee0872-450538b8217d40ce945d51e2c171ee36 received from {username='slave'} at 127.9.176.197:38695
I20240502 01:55:25.572774 25154 tablet_copy_service.cc:433] P 672a55e7d4874c5d99f3c5e3d9550f15: ending tablet copy session 6742b9db50c64ea394c2ca38ddee0872-450538b8217d40ce945d51e2c171ee36 on tablet 450538b8217d40ce945d51e2c171ee36 with peer 6742b9db50c64ea394c2ca38ddee0872
I20240502 01:55:25.575685 26556 tablet_bootstrap.cc:492] T 08bb5eb2c2814019b6eda6fbbc288245 P 30921110704f47f88d68a7d9364d42cc: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:25.576184 26556 tablet_bootstrap.cc:492] T 08bb5eb2c2814019b6eda6fbbc288245 P 30921110704f47f88d68a7d9364d42cc: Bootstrap complete.
I20240502 01:55:25.576535 26556 ts_tablet_manager.cc:1367] T 08bb5eb2c2814019b6eda6fbbc288245 P 30921110704f47f88d68a7d9364d42cc: Time spent bootstrapping tablet: real 0.222s	user 0.064s	sys 0.000s
I20240502 01:55:25.577510 26556 raft_consensus.cc:348] T 08bb5eb2c2814019b6eda6fbbc288245 P 30921110704f47f88d68a7d9364d42cc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } attrs { replace: true } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: true } }
I20240502 01:55:25.577742 26556 raft_consensus.cc:729] T 08bb5eb2c2814019b6eda6fbbc288245 P 30921110704f47f88d68a7d9364d42cc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 30921110704f47f88d68a7d9364d42cc, State: Initialized, Role: LEARNER
I20240502 01:55:25.577991 26556 consensus_queue.cc:260] T 08bb5eb2c2814019b6eda6fbbc288245 P 30921110704f47f88d68a7d9364d42cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } attrs { replace: true } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: true } }
I20240502 01:55:25.578531 26556 ts_tablet_manager.cc:1398] T 08bb5eb2c2814019b6eda6fbbc288245 P 30921110704f47f88d68a7d9364d42cc: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 01:55:25.579442 25022 tablet_copy_service.cc:342] P 75bd3f0e8d7c4e8aa92a353301afeb21: Request end of tablet copy session 30921110704f47f88d68a7d9364d42cc-08bb5eb2c2814019b6eda6fbbc288245 received from {username='slave'} at 127.9.176.196:56167
I20240502 01:55:25.579654 25022 tablet_copy_service.cc:433] P 75bd3f0e8d7c4e8aa92a353301afeb21: ending tablet copy session 30921110704f47f88d68a7d9364d42cc-08bb5eb2c2814019b6eda6fbbc288245 on tablet 08bb5eb2c2814019b6eda6fbbc288245 with peer 30921110704f47f88d68a7d9364d42cc
I20240502 01:55:25.582909 25934 raft_consensus.cc:1203] T 08bb5eb2c2814019b6eda6fbbc288245 P 30921110704f47f88d68a7d9364d42cc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 01:55:25.595958 26529 tablet_bootstrap.cc:492] T 32b0189cb19f42efbe7118003e9390f9 P 30921110704f47f88d68a7d9364d42cc: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:25.596516 26529 tablet_bootstrap.cc:492] T 32b0189cb19f42efbe7118003e9390f9 P 30921110704f47f88d68a7d9364d42cc: Bootstrap complete.
I20240502 01:55:25.596848 26529 ts_tablet_manager.cc:1367] T 32b0189cb19f42efbe7118003e9390f9 P 30921110704f47f88d68a7d9364d42cc: Time spent bootstrapping tablet: real 0.180s	user 0.045s	sys 0.009s
I20240502 01:55:25.597534 26529 raft_consensus.cc:348] T 32b0189cb19f42efbe7118003e9390f9 P 30921110704f47f88d68a7d9364d42cc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } attrs { replace: true } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: true } }
I20240502 01:55:25.597759 26529 raft_consensus.cc:729] T 32b0189cb19f42efbe7118003e9390f9 P 30921110704f47f88d68a7d9364d42cc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 30921110704f47f88d68a7d9364d42cc, State: Initialized, Role: LEARNER
I20240502 01:55:25.597962 26529 consensus_queue.cc:260] T 32b0189cb19f42efbe7118003e9390f9 P 30921110704f47f88d68a7d9364d42cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } attrs { replace: true } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: true } }
I20240502 01:55:25.598524 26529 ts_tablet_manager.cc:1398] T 32b0189cb19f42efbe7118003e9390f9 P 30921110704f47f88d68a7d9364d42cc: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 01:55:25.599396 24889 tablet_copy_service.cc:342] P 96e5c2fd5d624c2da735e72b387a12d9: Request end of tablet copy session 30921110704f47f88d68a7d9364d42cc-32b0189cb19f42efbe7118003e9390f9 received from {username='slave'} at 127.9.176.196:36527
I20240502 01:55:25.599545 24889 tablet_copy_service.cc:433] P 96e5c2fd5d624c2da735e72b387a12d9: ending tablet copy session 30921110704f47f88d68a7d9364d42cc-32b0189cb19f42efbe7118003e9390f9 on tablet 32b0189cb19f42efbe7118003e9390f9 with peer 30921110704f47f88d68a7d9364d42cc
I20240502 01:55:25.622733 25934 raft_consensus.cc:1203] T f77025fd71924964afafcd32c9ed6923 P 30921110704f47f88d68a7d9364d42cc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 01:55:25.670400 26066 raft_consensus.cc:1203] T ce37ee2e946b4075b78f3c9456939848 P d490b54f0e954ef59deb6750523f6027 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 01:55:25.712055 26197 raft_consensus.cc:1203] T 450538b8217d40ce945d51e2c171ee36 P 6742b9db50c64ea394c2ca38ddee0872 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 01:55:25.733753 25934 raft_consensus.cc:1203] T 32b0189cb19f42efbe7118003e9390f9 P 30921110704f47f88d68a7d9364d42cc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 01:55:25.774708 26197 raft_consensus.cc:1203] T b6beb3b47fd545d2b6a2d9181f4e5468 P 6742b9db50c64ea394c2ca38ddee0872 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 01:55:25.777141 26066 raft_consensus.cc:1203] T a9c57322de524bce8a114d9a94bd019d P d490b54f0e954ef59deb6750523f6027 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 01:55:25.875156 26197 raft_consensus.cc:1203] T 75d71cdcf4d44353acb7d2f18696977b P 6742b9db50c64ea394c2ca38ddee0872 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 01:55:25.915480 26613 raft_consensus.cc:1050] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 75bd3f0e8d7c4e8aa92a353301afeb21: attempting to promote NON_VOTER d490b54f0e954ef59deb6750523f6027 to VOTER
I20240502 01:55:25.916442 26613 consensus_queue.cc:237] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 75bd3f0e8d7c4e8aa92a353301afeb21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } attrs { replace: true } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: false } }
I20240502 01:55:25.919072 25133 raft_consensus.cc:1261] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 1 FOLLOWER]: Refusing update from remote peer 75bd3f0e8d7c4e8aa92a353301afeb21: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:25.919063 24868 raft_consensus.cc:1261] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 96e5c2fd5d624c2da735e72b387a12d9 [term 1 FOLLOWER]: Refusing update from remote peer 75bd3f0e8d7c4e8aa92a353301afeb21: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:25.919289 26066 raft_consensus.cc:1261] T 1b76f351e8644d9e8ff27e3f1a963ba2 P d490b54f0e954ef59deb6750523f6027 [term 1 LEARNER]: Refusing update from remote peer 75bd3f0e8d7c4e8aa92a353301afeb21: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:25.919775 26613 consensus_queue.cc:1035] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 75bd3f0e8d7c4e8aa92a353301afeb21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:25.920176 26396 consensus_queue.cc:1035] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 75bd3f0e8d7c4e8aa92a353301afeb21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:25.920493 26682 consensus_queue.cc:1035] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 75bd3f0e8d7c4e8aa92a353301afeb21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:25.924801 26682 raft_consensus.cc:2932] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d490b54f0e954ef59deb6750523f6027 (127.9.176.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } attrs { replace: true } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: false } } }
I20240502 01:55:25.925850 26066 raft_consensus.cc:2932] T 1b76f351e8644d9e8ff27e3f1a963ba2 P d490b54f0e954ef59deb6750523f6027 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d490b54f0e954ef59deb6750523f6027 (127.9.176.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } attrs { replace: true } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: false } } }
I20240502 01:55:25.929849 24868 raft_consensus.cc:2932] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 96e5c2fd5d624c2da735e72b387a12d9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d490b54f0e954ef59deb6750523f6027 (127.9.176.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } attrs { replace: true } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: false } } }
I20240502 01:55:25.931754 24725 catalog_manager.cc:5484] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 75bd3f0e8d7c4e8aa92a353301afeb21 reported cstate change: config changed from index 4 to 5, d490b54f0e954ef59deb6750523f6027 (127.9.176.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:55:25.935365 25133 raft_consensus.cc:2932] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d490b54f0e954ef59deb6750523f6027 (127.9.176.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } attrs { replace: true } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: false } } }
I20240502 01:55:25.943810 25002 consensus_queue.cc:237] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 75bd3f0e8d7c4e8aa92a353301afeb21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: false } }
I20240502 01:55:25.945840 26066 raft_consensus.cc:1261] T 1b76f351e8644d9e8ff27e3f1a963ba2 P d490b54f0e954ef59deb6750523f6027 [term 1 FOLLOWER]: Refusing update from remote peer 75bd3f0e8d7c4e8aa92a353301afeb21: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:25.945927 25133 raft_consensus.cc:1261] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 1 FOLLOWER]: Refusing update from remote peer 75bd3f0e8d7c4e8aa92a353301afeb21: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:25.946493 26614 consensus_queue.cc:1035] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 75bd3f0e8d7c4e8aa92a353301afeb21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:25.947011 26613 consensus_queue.cc:1035] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 75bd3f0e8d7c4e8aa92a353301afeb21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:25.949642 26396 raft_consensus.cc:2932] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 96e5c2fd5d624c2da735e72b387a12d9 (127.9.176.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: false } } }
I20240502 01:55:25.950695 26066 raft_consensus.cc:2932] T 1b76f351e8644d9e8ff27e3f1a963ba2 P d490b54f0e954ef59deb6750523f6027 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 96e5c2fd5d624c2da735e72b387a12d9 (127.9.176.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: false } } }
I20240502 01:55:25.950928 25133 raft_consensus.cc:2932] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 96e5c2fd5d624c2da735e72b387a12d9 (127.9.176.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: false } } }
I20240502 01:55:25.954232 24712 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1b76f351e8644d9e8ff27e3f1a963ba2 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:55:25.957098 24725 catalog_manager.cc:5484] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 75bd3f0e8d7c4e8aa92a353301afeb21 reported cstate change: config changed from index 5 to 6, VOTER 96e5c2fd5d624c2da735e72b387a12d9 (127.9.176.193) evicted. New cstate: current_term: 1 leader_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:55:25.961797 24848 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b76f351e8644d9e8ff27e3f1a963ba2 with delete_type TABLET_DATA_TOMBSTONED (TS 96e5c2fd5d624c2da735e72b387a12d9 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:54848
I20240502 01:55:25.963030 26690 tablet_replica.cc:330] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 96e5c2fd5d624c2da735e72b387a12d9: stopping tablet replica
I20240502 01:55:25.963670 26690 raft_consensus.cc:2229] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 96e5c2fd5d624c2da735e72b387a12d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:25.964125 26690 raft_consensus.cc:2258] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 96e5c2fd5d624c2da735e72b387a12d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:25.965641 26690 ts_tablet_manager.cc:1875] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 96e5c2fd5d624c2da735e72b387a12d9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:25.970664 26690 ts_tablet_manager.cc:1888] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 96e5c2fd5d624c2da735e72b387a12d9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:55:25.970834 26690 log.cc:1196] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 96e5c2fd5d624c2da735e72b387a12d9: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/wal/wals/1b76f351e8644d9e8ff27e3f1a963ba2
I20240502 01:55:25.971783 24712 catalog_manager.cc:4830] TS 96e5c2fd5d624c2da735e72b387a12d9 (127.9.176.193:32783): tablet 1b76f351e8644d9e8ff27e3f1a963ba2 (table rebalance_test_table_0 [id=a39bf554fb4649d8983f51465c4cd34d]) successfully deleted
I20240502 01:55:26.020586 26614 raft_consensus.cc:1050] T 08bb5eb2c2814019b6eda6fbbc288245 P 75bd3f0e8d7c4e8aa92a353301afeb21: attempting to promote NON_VOTER 30921110704f47f88d68a7d9364d42cc to VOTER
I20240502 01:55:26.021456 26614 consensus_queue.cc:237] T 08bb5eb2c2814019b6eda6fbbc288245 P 75bd3f0e8d7c4e8aa92a353301afeb21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } attrs { replace: true } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: false } }
I20240502 01:55:26.023844 24868 raft_consensus.cc:1261] T 08bb5eb2c2814019b6eda6fbbc288245 P 96e5c2fd5d624c2da735e72b387a12d9 [term 1 FOLLOWER]: Refusing update from remote peer 75bd3f0e8d7c4e8aa92a353301afeb21: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:26.023860 25133 raft_consensus.cc:1261] T 08bb5eb2c2814019b6eda6fbbc288245 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 1 FOLLOWER]: Refusing update from remote peer 75bd3f0e8d7c4e8aa92a353301afeb21: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:26.023937 25934 raft_consensus.cc:1261] T 08bb5eb2c2814019b6eda6fbbc288245 P 30921110704f47f88d68a7d9364d42cc [term 1 LEARNER]: Refusing update from remote peer 75bd3f0e8d7c4e8aa92a353301afeb21: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:26.024436 26396 consensus_queue.cc:1035] T 08bb5eb2c2814019b6eda6fbbc288245 P 75bd3f0e8d7c4e8aa92a353301afeb21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:26.024786 26613 consensus_queue.cc:1035] T 08bb5eb2c2814019b6eda6fbbc288245 P 75bd3f0e8d7c4e8aa92a353301afeb21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:26.025069 26682 consensus_queue.cc:1035] T 08bb5eb2c2814019b6eda6fbbc288245 P 75bd3f0e8d7c4e8aa92a353301afeb21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:26.028880 26613 raft_consensus.cc:2932] T 08bb5eb2c2814019b6eda6fbbc288245 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 30921110704f47f88d68a7d9364d42cc (127.9.176.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } attrs { replace: true } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: false } } }
I20240502 01:55:26.029470 24868 raft_consensus.cc:2932] T 08bb5eb2c2814019b6eda6fbbc288245 P 96e5c2fd5d624c2da735e72b387a12d9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 30921110704f47f88d68a7d9364d42cc (127.9.176.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } attrs { replace: true } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: false } } }
I20240502 01:55:26.034008 24725 catalog_manager.cc:5484] T 08bb5eb2c2814019b6eda6fbbc288245 P 96e5c2fd5d624c2da735e72b387a12d9 reported cstate change: config changed from index 4 to 5, 30921110704f47f88d68a7d9364d42cc (127.9.176.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } attrs { replace: true } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: false } } }
I20240502 01:55:26.035426 25133 raft_consensus.cc:2932] T 08bb5eb2c2814019b6eda6fbbc288245 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 30921110704f47f88d68a7d9364d42cc (127.9.176.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } attrs { replace: true } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: false } } }
I20240502 01:55:26.040946 25934 raft_consensus.cc:2932] T 08bb5eb2c2814019b6eda6fbbc288245 P 30921110704f47f88d68a7d9364d42cc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 30921110704f47f88d68a7d9364d42cc (127.9.176.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } attrs { replace: true } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: false } } }
I20240502 01:55:26.042934 25002 consensus_queue.cc:237] T 08bb5eb2c2814019b6eda6fbbc288245 P 75bd3f0e8d7c4e8aa92a353301afeb21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: false } }
I20240502 01:55:26.044956 25933 raft_consensus.cc:1261] T 08bb5eb2c2814019b6eda6fbbc288245 P 30921110704f47f88d68a7d9364d42cc [term 1 FOLLOWER]: Refusing update from remote peer 75bd3f0e8d7c4e8aa92a353301afeb21: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:26.045514 26614 consensus_queue.cc:1035] T 08bb5eb2c2814019b6eda6fbbc288245 P 75bd3f0e8d7c4e8aa92a353301afeb21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:26.046334 24868 raft_consensus.cc:1261] T 08bb5eb2c2814019b6eda6fbbc288245 P 96e5c2fd5d624c2da735e72b387a12d9 [term 1 FOLLOWER]: Refusing update from remote peer 75bd3f0e8d7c4e8aa92a353301afeb21: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:26.047837 26682 consensus_queue.cc:1035] T 08bb5eb2c2814019b6eda6fbbc288245 P 75bd3f0e8d7c4e8aa92a353301afeb21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:26.048346 26613 raft_consensus.cc:2932] T 08bb5eb2c2814019b6eda6fbbc288245 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 672a55e7d4874c5d99f3c5e3d9550f15 (127.9.176.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: false } } }
I20240502 01:55:26.049018 25933 raft_consensus.cc:2932] T 08bb5eb2c2814019b6eda6fbbc288245 P 30921110704f47f88d68a7d9364d42cc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 672a55e7d4874c5d99f3c5e3d9550f15 (127.9.176.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: false } } }
I20240502 01:55:26.050895 24868 raft_consensus.cc:2932] T 08bb5eb2c2814019b6eda6fbbc288245 P 96e5c2fd5d624c2da735e72b387a12d9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 672a55e7d4874c5d99f3c5e3d9550f15 (127.9.176.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: false } } }
I20240502 01:55:26.053506 24712 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 08bb5eb2c2814019b6eda6fbbc288245 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:55:26.054698 24724 catalog_manager.cc:5484] T 08bb5eb2c2814019b6eda6fbbc288245 P 75bd3f0e8d7c4e8aa92a353301afeb21 reported cstate change: config changed from index 5 to 6, VOTER 672a55e7d4874c5d99f3c5e3d9550f15 (127.9.176.195) evicted. New cstate: current_term: 1 leader_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:55:26.058874 25112 tablet_service.cc:1514] Processing DeleteTablet for tablet 08bb5eb2c2814019b6eda6fbbc288245 with delete_type TABLET_DATA_TOMBSTONED (TS 672a55e7d4874c5d99f3c5e3d9550f15 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:46766
I20240502 01:55:26.059998 26695 tablet_replica.cc:330] T 08bb5eb2c2814019b6eda6fbbc288245 P 672a55e7d4874c5d99f3c5e3d9550f15: stopping tablet replica
I20240502 01:55:26.060640 26695 raft_consensus.cc:2229] T 08bb5eb2c2814019b6eda6fbbc288245 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:26.061138 26695 raft_consensus.cc:2258] T 08bb5eb2c2814019b6eda6fbbc288245 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:26.062584 26695 ts_tablet_manager.cc:1875] T 08bb5eb2c2814019b6eda6fbbc288245 P 672a55e7d4874c5d99f3c5e3d9550f15: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:26.067350 26695 ts_tablet_manager.cc:1888] T 08bb5eb2c2814019b6eda6fbbc288245 P 672a55e7d4874c5d99f3c5e3d9550f15: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:55:26.067536 26695 log.cc:1196] T 08bb5eb2c2814019b6eda6fbbc288245 P 672a55e7d4874c5d99f3c5e3d9550f15: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/wal/wals/08bb5eb2c2814019b6eda6fbbc288245
I20240502 01:55:26.068497 24712 catalog_manager.cc:4830] TS 672a55e7d4874c5d99f3c5e3d9550f15 (127.9.176.195:46579): tablet 08bb5eb2c2814019b6eda6fbbc288245 (table rebalance_test_table_2 [id=26fb053e91c344cab36e6f0312de1a3c]) successfully deleted
I20240502 01:55:26.181448 26632 raft_consensus.cc:1050] T ce37ee2e946b4075b78f3c9456939848 P 672a55e7d4874c5d99f3c5e3d9550f15: attempting to promote NON_VOTER d490b54f0e954ef59deb6750523f6027 to VOTER
I20240502 01:55:26.182657 26632 consensus_queue.cc:237] T ce37ee2e946b4075b78f3c9456939848 P 672a55e7d4874c5d99f3c5e3d9550f15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } attrs { replace: true } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: false } }
I20240502 01:55:26.189456 25002 raft_consensus.cc:1261] T ce37ee2e946b4075b78f3c9456939848 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 1 FOLLOWER]: Refusing update from remote peer 672a55e7d4874c5d99f3c5e3d9550f15: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:26.190652 24868 raft_consensus.cc:1261] T ce37ee2e946b4075b78f3c9456939848 P 96e5c2fd5d624c2da735e72b387a12d9 [term 1 FOLLOWER]: Refusing update from remote peer 672a55e7d4874c5d99f3c5e3d9550f15: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:26.190896 26066 raft_consensus.cc:1261] T ce37ee2e946b4075b78f3c9456939848 P d490b54f0e954ef59deb6750523f6027 [term 1 LEARNER]: Refusing update from remote peer 672a55e7d4874c5d99f3c5e3d9550f15: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:26.190294 26632 consensus_queue.cc:1035] T ce37ee2e946b4075b78f3c9456939848 P 672a55e7d4874c5d99f3c5e3d9550f15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.002s
I20240502 01:55:26.192145 26631 consensus_queue.cc:1035] T ce37ee2e946b4075b78f3c9456939848 P 672a55e7d4874c5d99f3c5e3d9550f15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:26.193018 26439 consensus_queue.cc:1035] T ce37ee2e946b4075b78f3c9456939848 P 672a55e7d4874c5d99f3c5e3d9550f15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:26.201278 26631 raft_consensus.cc:2932] T ce37ee2e946b4075b78f3c9456939848 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d490b54f0e954ef59deb6750523f6027 (127.9.176.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } attrs { replace: true } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: false } } }
I20240502 01:55:26.205727 24868 raft_consensus.cc:2932] T ce37ee2e946b4075b78f3c9456939848 P 96e5c2fd5d624c2da735e72b387a12d9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d490b54f0e954ef59deb6750523f6027 (127.9.176.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } attrs { replace: true } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: false } } }
I20240502 01:55:26.212052 26066 raft_consensus.cc:2932] T ce37ee2e946b4075b78f3c9456939848 P d490b54f0e954ef59deb6750523f6027 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d490b54f0e954ef59deb6750523f6027 (127.9.176.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } attrs { replace: true } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: false } } }
I20240502 01:55:26.216643 24722 catalog_manager.cc:5484] T ce37ee2e946b4075b78f3c9456939848 P 672a55e7d4874c5d99f3c5e3d9550f15 reported cstate change: config changed from index 4 to 5, d490b54f0e954ef59deb6750523f6027 (127.9.176.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:55:26.218796 25002 raft_consensus.cc:2932] T ce37ee2e946b4075b78f3c9456939848 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d490b54f0e954ef59deb6750523f6027 (127.9.176.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } attrs { replace: true } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: false } } }
I20240502 01:55:26.225893 25133 consensus_queue.cc:237] T ce37ee2e946b4075b78f3c9456939848 P 672a55e7d4874c5d99f3c5e3d9550f15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: false } }
I20240502 01:55:26.227836 26066 raft_consensus.cc:1261] T ce37ee2e946b4075b78f3c9456939848 P d490b54f0e954ef59deb6750523f6027 [term 1 FOLLOWER]: Refusing update from remote peer 672a55e7d4874c5d99f3c5e3d9550f15: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:26.227905 25002 raft_consensus.cc:1261] T ce37ee2e946b4075b78f3c9456939848 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 1 FOLLOWER]: Refusing update from remote peer 672a55e7d4874c5d99f3c5e3d9550f15: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:26.228525 26631 consensus_queue.cc:1035] T ce37ee2e946b4075b78f3c9456939848 P 672a55e7d4874c5d99f3c5e3d9550f15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:26.228889 26632 consensus_queue.cc:1035] T ce37ee2e946b4075b78f3c9456939848 P 672a55e7d4874c5d99f3c5e3d9550f15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:26.231905 26631 raft_consensus.cc:2932] T ce37ee2e946b4075b78f3c9456939848 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 96e5c2fd5d624c2da735e72b387a12d9 (127.9.176.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: false } } }
I20240502 01:55:26.235414 26614 raft_consensus.cc:1050] T f77025fd71924964afafcd32c9ed6923 P 75bd3f0e8d7c4e8aa92a353301afeb21: attempting to promote NON_VOTER 30921110704f47f88d68a7d9364d42cc to VOTER
I20240502 01:55:26.236268 24712 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ce37ee2e946b4075b78f3c9456939848 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:55:26.236409 26614 consensus_queue.cc:237] T f77025fd71924964afafcd32c9ed6923 P 75bd3f0e8d7c4e8aa92a353301afeb21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } attrs { replace: true } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: false } }
I20240502 01:55:26.238353 26066 raft_consensus.cc:2932] T ce37ee2e946b4075b78f3c9456939848 P d490b54f0e954ef59deb6750523f6027 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 96e5c2fd5d624c2da735e72b387a12d9 (127.9.176.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: false } } }
I20240502 01:55:26.239379 25002 raft_consensus.cc:2932] T ce37ee2e946b4075b78f3c9456939848 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 96e5c2fd5d624c2da735e72b387a12d9 (127.9.176.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: false } } }
I20240502 01:55:26.245340 24724 catalog_manager.cc:5484] T ce37ee2e946b4075b78f3c9456939848 P 672a55e7d4874c5d99f3c5e3d9550f15 reported cstate change: config changed from index 5 to 6, VOTER 96e5c2fd5d624c2da735e72b387a12d9 (127.9.176.193) evicted. New cstate: current_term: 1 leader_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:55:26.246484 25933 raft_consensus.cc:1261] T f77025fd71924964afafcd32c9ed6923 P 30921110704f47f88d68a7d9364d42cc [term 1 LEARNER]: Refusing update from remote peer 75bd3f0e8d7c4e8aa92a353301afeb21: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:26.247620 26613 consensus_queue.cc:1035] T f77025fd71924964afafcd32c9ed6923 P 75bd3f0e8d7c4e8aa92a353301afeb21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:26.251159 24868 raft_consensus.cc:1261] T f77025fd71924964afafcd32c9ed6923 P 96e5c2fd5d624c2da735e72b387a12d9 [term 1 FOLLOWER]: Refusing update from remote peer 75bd3f0e8d7c4e8aa92a353301afeb21: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:26.251788 24848 tablet_service.cc:1514] Processing DeleteTablet for tablet ce37ee2e946b4075b78f3c9456939848 with delete_type TABLET_DATA_TOMBSTONED (TS 96e5c2fd5d624c2da735e72b387a12d9 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:54848
I20240502 01:55:26.252207 26690 tablet_replica.cc:330] T ce37ee2e946b4075b78f3c9456939848 P 96e5c2fd5d624c2da735e72b387a12d9: stopping tablet replica
I20240502 01:55:26.252403 26690 raft_consensus.cc:2229] T ce37ee2e946b4075b78f3c9456939848 P 96e5c2fd5d624c2da735e72b387a12d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:26.252748 26690 raft_consensus.cc:2258] T ce37ee2e946b4075b78f3c9456939848 P 96e5c2fd5d624c2da735e72b387a12d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:26.253166 26396 consensus_queue.cc:1035] T f77025fd71924964afafcd32c9ed6923 P 75bd3f0e8d7c4e8aa92a353301afeb21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:26.256300 26690 ts_tablet_manager.cc:1875] T ce37ee2e946b4075b78f3c9456939848 P 96e5c2fd5d624c2da735e72b387a12d9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:26.264845 26682 raft_consensus.cc:2932] T f77025fd71924964afafcd32c9ed6923 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 30921110704f47f88d68a7d9364d42cc (127.9.176.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } attrs { replace: true } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: false } } }
I20240502 01:55:26.265583 25133 raft_consensus.cc:1261] T f77025fd71924964afafcd32c9ed6923 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 1 FOLLOWER]: Refusing update from remote peer 75bd3f0e8d7c4e8aa92a353301afeb21: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:26.266475 26613 consensus_queue.cc:1035] T f77025fd71924964afafcd32c9ed6923 P 75bd3f0e8d7c4e8aa92a353301afeb21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:26.265959 25933 raft_consensus.cc:2932] T f77025fd71924964afafcd32c9ed6923 P 30921110704f47f88d68a7d9364d42cc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 30921110704f47f88d68a7d9364d42cc (127.9.176.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } attrs { replace: true } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: false } } }
I20240502 01:55:26.270736 24868 raft_consensus.cc:2932] T f77025fd71924964afafcd32c9ed6923 P 96e5c2fd5d624c2da735e72b387a12d9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 30921110704f47f88d68a7d9364d42cc (127.9.176.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } attrs { replace: true } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: false } } }
I20240502 01:55:26.271317 24724 catalog_manager.cc:5484] T f77025fd71924964afafcd32c9ed6923 P 75bd3f0e8d7c4e8aa92a353301afeb21 reported cstate change: config changed from index 4 to 5, 30921110704f47f88d68a7d9364d42cc (127.9.176.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:55:26.277596 26690 ts_tablet_manager.cc:1888] T ce37ee2e946b4075b78f3c9456939848 P 96e5c2fd5d624c2da735e72b387a12d9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:55:26.277778 26690 log.cc:1196] T ce37ee2e946b4075b78f3c9456939848 P 96e5c2fd5d624c2da735e72b387a12d9: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/wal/wals/ce37ee2e946b4075b78f3c9456939848
I20240502 01:55:26.278785 24712 catalog_manager.cc:4830] TS 96e5c2fd5d624c2da735e72b387a12d9 (127.9.176.193:32783): tablet ce37ee2e946b4075b78f3c9456939848 (table rebalance_test_table_0 [id=a39bf554fb4649d8983f51465c4cd34d]) successfully deleted
I20240502 01:55:26.279117 25133 raft_consensus.cc:2932] T f77025fd71924964afafcd32c9ed6923 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 30921110704f47f88d68a7d9364d42cc (127.9.176.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } attrs { replace: true } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: false } } }
I20240502 01:55:26.284776 25002 consensus_queue.cc:237] T f77025fd71924964afafcd32c9ed6923 P 75bd3f0e8d7c4e8aa92a353301afeb21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: false } }
I20240502 01:55:26.286727 24868 raft_consensus.cc:1261] T f77025fd71924964afafcd32c9ed6923 P 96e5c2fd5d624c2da735e72b387a12d9 [term 1 FOLLOWER]: Refusing update from remote peer 75bd3f0e8d7c4e8aa92a353301afeb21: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:26.287292 25933 raft_consensus.cc:1261] T f77025fd71924964afafcd32c9ed6923 P 30921110704f47f88d68a7d9364d42cc [term 1 FOLLOWER]: Refusing update from remote peer 75bd3f0e8d7c4e8aa92a353301afeb21: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:26.287326 26682 consensus_queue.cc:1035] T f77025fd71924964afafcd32c9ed6923 P 75bd3f0e8d7c4e8aa92a353301afeb21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:26.287984 26682 consensus_queue.cc:1035] T f77025fd71924964afafcd32c9ed6923 P 75bd3f0e8d7c4e8aa92a353301afeb21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:26.289898 26614 raft_consensus.cc:2932] T f77025fd71924964afafcd32c9ed6923 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 672a55e7d4874c5d99f3c5e3d9550f15 (127.9.176.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: false } } }
I20240502 01:55:26.290794 24868 raft_consensus.cc:2932] T f77025fd71924964afafcd32c9ed6923 P 96e5c2fd5d624c2da735e72b387a12d9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 672a55e7d4874c5d99f3c5e3d9550f15 (127.9.176.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: false } } }
I20240502 01:55:26.296550 24725 catalog_manager.cc:5484] T f77025fd71924964afafcd32c9ed6923 P 96e5c2fd5d624c2da735e72b387a12d9 reported cstate change: config changed from index 5 to 6, VOTER 672a55e7d4874c5d99f3c5e3d9550f15 (127.9.176.195) evicted. New cstate: current_term: 1 leader_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: false } } }
I20240502 01:55:26.298236 24712 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f77025fd71924964afafcd32c9ed6923 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:55:26.298691 25933 raft_consensus.cc:2932] T f77025fd71924964afafcd32c9ed6923 P 30921110704f47f88d68a7d9364d42cc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 672a55e7d4874c5d99f3c5e3d9550f15 (127.9.176.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: false } } }
I20240502 01:55:26.301381 25112 tablet_service.cc:1514] Processing DeleteTablet for tablet f77025fd71924964afafcd32c9ed6923 with delete_type TABLET_DATA_TOMBSTONED (TS 672a55e7d4874c5d99f3c5e3d9550f15 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:46766
I20240502 01:55:26.301904 26695 tablet_replica.cc:330] T f77025fd71924964afafcd32c9ed6923 P 672a55e7d4874c5d99f3c5e3d9550f15: stopping tablet replica
I20240502 01:55:26.302168 26695 raft_consensus.cc:2229] T f77025fd71924964afafcd32c9ed6923 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:26.302440 26695 raft_consensus.cc:2258] T f77025fd71924964afafcd32c9ed6923 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:26.303438 26695 ts_tablet_manager.cc:1875] T f77025fd71924964afafcd32c9ed6923 P 672a55e7d4874c5d99f3c5e3d9550f15: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:26.308564 26695 ts_tablet_manager.cc:1888] T f77025fd71924964afafcd32c9ed6923 P 672a55e7d4874c5d99f3c5e3d9550f15: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:55:26.308696 26695 log.cc:1196] T f77025fd71924964afafcd32c9ed6923 P 672a55e7d4874c5d99f3c5e3d9550f15: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/wal/wals/f77025fd71924964afafcd32c9ed6923
I20240502 01:55:26.309444 24712 catalog_manager.cc:4830] TS 672a55e7d4874c5d99f3c5e3d9550f15 (127.9.176.195:46579): tablet f77025fd71924964afafcd32c9ed6923 (table rebalance_test_table_0 [id=a39bf554fb4649d8983f51465c4cd34d]) successfully deleted
I20240502 01:55:26.316483 26632 raft_consensus.cc:1050] T a9c57322de524bce8a114d9a94bd019d P 672a55e7d4874c5d99f3c5e3d9550f15: attempting to promote NON_VOTER d490b54f0e954ef59deb6750523f6027 to VOTER
I20240502 01:55:26.317440 26632 consensus_queue.cc:237] T a9c57322de524bce8a114d9a94bd019d P 672a55e7d4874c5d99f3c5e3d9550f15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } attrs { replace: true } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: false } }
I20240502 01:55:26.320181 26066 raft_consensus.cc:1261] T a9c57322de524bce8a114d9a94bd019d P d490b54f0e954ef59deb6750523f6027 [term 1 LEARNER]: Refusing update from remote peer 672a55e7d4874c5d99f3c5e3d9550f15: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:26.321731 26439 consensus_queue.cc:1035] T a9c57322de524bce8a114d9a94bd019d P 672a55e7d4874c5d99f3c5e3d9550f15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:26.324040 26633 raft_consensus.cc:1050] T 32b0189cb19f42efbe7118003e9390f9 P 96e5c2fd5d624c2da735e72b387a12d9: attempting to promote NON_VOTER 30921110704f47f88d68a7d9364d42cc to VOTER
I20240502 01:55:26.325495 26633 consensus_queue.cc:237] T 32b0189cb19f42efbe7118003e9390f9 P 96e5c2fd5d624c2da735e72b387a12d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } attrs { replace: true } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: false } }
I20240502 01:55:26.328332 25002 raft_consensus.cc:1261] T a9c57322de524bce8a114d9a94bd019d P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 1 FOLLOWER]: Refusing update from remote peer 672a55e7d4874c5d99f3c5e3d9550f15: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:26.328897 24868 raft_consensus.cc:1261] T a9c57322de524bce8a114d9a94bd019d P 96e5c2fd5d624c2da735e72b387a12d9 [term 1 FOLLOWER]: Refusing update from remote peer 672a55e7d4874c5d99f3c5e3d9550f15: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:26.329437 26766 consensus_queue.cc:1035] T a9c57322de524bce8a114d9a94bd019d P 672a55e7d4874c5d99f3c5e3d9550f15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:26.330703 25133 raft_consensus.cc:1261] T 32b0189cb19f42efbe7118003e9390f9 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 1 FOLLOWER]: Refusing update from remote peer 96e5c2fd5d624c2da735e72b387a12d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:26.332020 26634 consensus_queue.cc:1035] T 32b0189cb19f42efbe7118003e9390f9 P 96e5c2fd5d624c2da735e72b387a12d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:26.332772 26631 consensus_queue.cc:1035] T a9c57322de524bce8a114d9a94bd019d P 672a55e7d4874c5d99f3c5e3d9550f15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:26.337265 26766 raft_consensus.cc:1050] T 450538b8217d40ce945d51e2c171ee36 P 672a55e7d4874c5d99f3c5e3d9550f15: attempting to promote NON_VOTER 6742b9db50c64ea394c2ca38ddee0872 to VOTER
I20240502 01:55:26.338475 26766 consensus_queue.cc:237] T 450538b8217d40ce945d51e2c171ee36 P 672a55e7d4874c5d99f3c5e3d9550f15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { replace: true } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false } }
I20240502 01:55:26.340544 25933 raft_consensus.cc:1261] T 32b0189cb19f42efbe7118003e9390f9 P 30921110704f47f88d68a7d9364d42cc [term 1 LEARNER]: Refusing update from remote peer 96e5c2fd5d624c2da735e72b387a12d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:26.341357 26634 consensus_queue.cc:1035] T 32b0189cb19f42efbe7118003e9390f9 P 96e5c2fd5d624c2da735e72b387a12d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:26.341734 26632 raft_consensus.cc:2932] T a9c57322de524bce8a114d9a94bd019d P 672a55e7d4874c5d99f3c5e3d9550f15 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d490b54f0e954ef59deb6750523f6027 (127.9.176.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } attrs { replace: true } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: false } } }
I20240502 01:55:26.343271 26439 raft_consensus.cc:1050] T b6beb3b47fd545d2b6a2d9181f4e5468 P 672a55e7d4874c5d99f3c5e3d9550f15: attempting to promote NON_VOTER 6742b9db50c64ea394c2ca38ddee0872 to VOTER
I20240502 01:55:26.343611 26066 raft_consensus.cc:2932] T a9c57322de524bce8a114d9a94bd019d P d490b54f0e954ef59deb6750523f6027 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d490b54f0e954ef59deb6750523f6027 (127.9.176.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } attrs { replace: true } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: false } } }
I20240502 01:55:26.344292 26439 consensus_queue.cc:237] T b6beb3b47fd545d2b6a2d9181f4e5468 P 672a55e7d4874c5d99f3c5e3d9550f15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { replace: true } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false } }
I20240502 01:55:26.347306 25001 raft_consensus.cc:1261] T 32b0189cb19f42efbe7118003e9390f9 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 1 FOLLOWER]: Refusing update from remote peer 96e5c2fd5d624c2da735e72b387a12d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:26.347251 25002 raft_consensus.cc:2932] T a9c57322de524bce8a114d9a94bd019d P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d490b54f0e954ef59deb6750523f6027 (127.9.176.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } attrs { replace: true } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: false } } }
I20240502 01:55:26.348203 26634 consensus_queue.cc:1035] T 32b0189cb19f42efbe7118003e9390f9 P 96e5c2fd5d624c2da735e72b387a12d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:26.351560 24868 raft_consensus.cc:2932] T a9c57322de524bce8a114d9a94bd019d P 96e5c2fd5d624c2da735e72b387a12d9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d490b54f0e954ef59deb6750523f6027 (127.9.176.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } attrs { replace: true } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: false } } }
I20240502 01:55:26.354661 24722 catalog_manager.cc:5484] T a9c57322de524bce8a114d9a94bd019d P d490b54f0e954ef59deb6750523f6027 reported cstate change: config changed from index 4 to 5, d490b54f0e954ef59deb6750523f6027 (127.9.176.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } attrs { replace: true } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: false } } }
I20240502 01:55:26.372483 25134 consensus_queue.cc:237] T a9c57322de524bce8a114d9a94bd019d P 672a55e7d4874c5d99f3c5e3d9550f15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: false } }
I20240502 01:55:26.375448 25002 raft_consensus.cc:1261] T a9c57322de524bce8a114d9a94bd019d P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 1 FOLLOWER]: Refusing update from remote peer 672a55e7d4874c5d99f3c5e3d9550f15: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:26.376232 26632 consensus_queue.cc:1035] T a9c57322de524bce8a114d9a94bd019d P 672a55e7d4874c5d99f3c5e3d9550f15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:26.378011 26066 raft_consensus.cc:1261] T a9c57322de524bce8a114d9a94bd019d P d490b54f0e954ef59deb6750523f6027 [term 1 FOLLOWER]: Refusing update from remote peer 672a55e7d4874c5d99f3c5e3d9550f15: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:26.378935 26632 consensus_queue.cc:1035] T a9c57322de524bce8a114d9a94bd019d P 672a55e7d4874c5d99f3c5e3d9550f15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:26.379629 26631 raft_consensus.cc:2932] T a9c57322de524bce8a114d9a94bd019d P 672a55e7d4874c5d99f3c5e3d9550f15 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 96e5c2fd5d624c2da735e72b387a12d9 (127.9.176.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: false } } }
I20240502 01:55:26.386694 24712 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a9c57322de524bce8a114d9a94bd019d with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:55:26.388528 26066 raft_consensus.cc:2932] T a9c57322de524bce8a114d9a94bd019d P d490b54f0e954ef59deb6750523f6027 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 96e5c2fd5d624c2da735e72b387a12d9 (127.9.176.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: false } } }
I20240502 01:55:26.388758 25002 raft_consensus.cc:2932] T a9c57322de524bce8a114d9a94bd019d P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 96e5c2fd5d624c2da735e72b387a12d9 (127.9.176.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: false } } }
I20240502 01:55:26.390376 26634 raft_consensus.cc:2932] T 32b0189cb19f42efbe7118003e9390f9 P 96e5c2fd5d624c2da735e72b387a12d9 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 30921110704f47f88d68a7d9364d42cc (127.9.176.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } attrs { replace: true } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: false } } }
I20240502 01:55:26.394629 24722 catalog_manager.cc:5484] T a9c57322de524bce8a114d9a94bd019d P 672a55e7d4874c5d99f3c5e3d9550f15 reported cstate change: config changed from index 5 to 6, VOTER 96e5c2fd5d624c2da735e72b387a12d9 (127.9.176.193) evicted. New cstate: current_term: 1 leader_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:55:26.395447 25000 raft_consensus.cc:2932] T 32b0189cb19f42efbe7118003e9390f9 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 30921110704f47f88d68a7d9364d42cc (127.9.176.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } attrs { replace: true } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: false } } }
I20240502 01:55:26.399314 24848 tablet_service.cc:1514] Processing DeleteTablet for tablet a9c57322de524bce8a114d9a94bd019d with delete_type TABLET_DATA_TOMBSTONED (TS 96e5c2fd5d624c2da735e72b387a12d9 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:54848
I20240502 01:55:26.399922 26690 tablet_replica.cc:330] T a9c57322de524bce8a114d9a94bd019d P 96e5c2fd5d624c2da735e72b387a12d9: stopping tablet replica
I20240502 01:55:26.400175 26690 raft_consensus.cc:2229] T a9c57322de524bce8a114d9a94bd019d P 96e5c2fd5d624c2da735e72b387a12d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:26.400537 26690 raft_consensus.cc:2258] T a9c57322de524bce8a114d9a94bd019d P 96e5c2fd5d624c2da735e72b387a12d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:26.403301 26690 ts_tablet_manager.cc:1875] T a9c57322de524bce8a114d9a94bd019d P 96e5c2fd5d624c2da735e72b387a12d9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:26.405968 26197 raft_consensus.cc:1261] T 450538b8217d40ce945d51e2c171ee36 P 6742b9db50c64ea394c2ca38ddee0872 [term 1 LEARNER]: Refusing update from remote peer 672a55e7d4874c5d99f3c5e3d9550f15: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:26.407526 26632 consensus_queue.cc:1035] T 450538b8217d40ce945d51e2c171ee36 P 672a55e7d4874c5d99f3c5e3d9550f15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:26.408839 24725 catalog_manager.cc:5484] T 32b0189cb19f42efbe7118003e9390f9 P 96e5c2fd5d624c2da735e72b387a12d9 reported cstate change: config changed from index 4 to 5, 30921110704f47f88d68a7d9364d42cc (127.9.176.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "96e5c2fd5d624c2da735e72b387a12d9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:55:26.409682 24869 raft_consensus.cc:1261] T b6beb3b47fd545d2b6a2d9181f4e5468 P 96e5c2fd5d624c2da735e72b387a12d9 [term 1 FOLLOWER]: Refusing update from remote peer 672a55e7d4874c5d99f3c5e3d9550f15: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:26.410409 26631 consensus_queue.cc:1035] T b6beb3b47fd545d2b6a2d9181f4e5468 P 672a55e7d4874c5d99f3c5e3d9550f15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:26.411020 26197 raft_consensus.cc:1261] T b6beb3b47fd545d2b6a2d9181f4e5468 P 6742b9db50c64ea394c2ca38ddee0872 [term 1 LEARNER]: Refusing update from remote peer 672a55e7d4874c5d99f3c5e3d9550f15: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:26.412596 25133 raft_consensus.cc:2932] T 32b0189cb19f42efbe7118003e9390f9 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 30921110704f47f88d68a7d9364d42cc (127.9.176.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } attrs { replace: true } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: false } } }
I20240502 01:55:26.412096 26631 consensus_queue.cc:1035] T b6beb3b47fd545d2b6a2d9181f4e5468 P 672a55e7d4874c5d99f3c5e3d9550f15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:26.416608 24867 consensus_queue.cc:237] T 32b0189cb19f42efbe7118003e9390f9 P 96e5c2fd5d624c2da735e72b387a12d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: false } }
I20240502 01:55:26.417613 25000 raft_consensus.cc:1261] T 450538b8217d40ce945d51e2c171ee36 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 1 FOLLOWER]: Refusing update from remote peer 672a55e7d4874c5d99f3c5e3d9550f15: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:26.421195 25001 raft_consensus.cc:1261] T b6beb3b47fd545d2b6a2d9181f4e5468 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 1 FOLLOWER]: Refusing update from remote peer 672a55e7d4874c5d99f3c5e3d9550f15: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:26.421798 25934 raft_consensus.cc:1261] T 32b0189cb19f42efbe7118003e9390f9 P 30921110704f47f88d68a7d9364d42cc [term 1 FOLLOWER]: Refusing update from remote peer 96e5c2fd5d624c2da735e72b387a12d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:26.422557 25001 raft_consensus.cc:1261] T 32b0189cb19f42efbe7118003e9390f9 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 1 FOLLOWER]: Refusing update from remote peer 96e5c2fd5d624c2da735e72b387a12d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:26.422691 26766 consensus_queue.cc:1035] T 450538b8217d40ce945d51e2c171ee36 P 672a55e7d4874c5d99f3c5e3d9550f15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240502 01:55:26.424000 26780 consensus_queue.cc:1035] T b6beb3b47fd545d2b6a2d9181f4e5468 P 672a55e7d4874c5d99f3c5e3d9550f15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:26.424872 24868 raft_consensus.cc:1261] T 450538b8217d40ce945d51e2c171ee36 P 96e5c2fd5d624c2da735e72b387a12d9 [term 1 FOLLOWER]: Refusing update from remote peer 672a55e7d4874c5d99f3c5e3d9550f15: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:26.424834 26782 consensus_queue.cc:1035] T 32b0189cb19f42efbe7118003e9390f9 P 96e5c2fd5d624c2da735e72b387a12d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:26.425305 26782 consensus_queue.cc:1035] T 32b0189cb19f42efbe7118003e9390f9 P 96e5c2fd5d624c2da735e72b387a12d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:26.430724 26439 consensus_queue.cc:1035] T 450538b8217d40ce945d51e2c171ee36 P 672a55e7d4874c5d99f3c5e3d9550f15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:26.431556 25934 raft_consensus.cc:2932] T 32b0189cb19f42efbe7118003e9390f9 P 30921110704f47f88d68a7d9364d42cc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 30921110704f47f88d68a7d9364d42cc (127.9.176.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } attrs { replace: true } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: false } } }
I20240502 01:55:26.438865 26780 raft_consensus.cc:2932] T 450538b8217d40ce945d51e2c171ee36 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6742b9db50c64ea394c2ca38ddee0872 (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { replace: true } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false } } }
I20240502 01:55:26.439254 26766 raft_consensus.cc:2932] T b6beb3b47fd545d2b6a2d9181f4e5468 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6742b9db50c64ea394c2ca38ddee0872 (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { replace: true } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false } } }
I20240502 01:55:26.444541 24722 catalog_manager.cc:5484] T 450538b8217d40ce945d51e2c171ee36 P 672a55e7d4874c5d99f3c5e3d9550f15 reported cstate change: config changed from index 4 to 5, 6742b9db50c64ea394c2ca38ddee0872 (127.9.176.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:55:26.444471 25000 raft_consensus.cc:2932] T b6beb3b47fd545d2b6a2d9181f4e5468 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6742b9db50c64ea394c2ca38ddee0872 (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { replace: true } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false } } }
I20240502 01:55:26.445901 26633 raft_consensus.cc:2932] T 32b0189cb19f42efbe7118003e9390f9 P 96e5c2fd5d624c2da735e72b387a12d9 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 672a55e7d4874c5d99f3c5e3d9550f15 (127.9.176.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: false } } }
I20240502 01:55:26.449733 25002 raft_consensus.cc:2932] T 32b0189cb19f42efbe7118003e9390f9 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 672a55e7d4874c5d99f3c5e3d9550f15 (127.9.176.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: false } } }
I20240502 01:55:26.452718 25000 raft_consensus.cc:2932] T 450538b8217d40ce945d51e2c171ee36 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6742b9db50c64ea394c2ca38ddee0872 (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { replace: true } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false } } }
I20240502 01:55:26.458364 24724 catalog_manager.cc:5484] T b6beb3b47fd545d2b6a2d9181f4e5468 P 672a55e7d4874c5d99f3c5e3d9550f15 reported cstate change: config changed from index 4 to 5, 6742b9db50c64ea394c2ca38ddee0872 (127.9.176.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:55:26.460059 24868 raft_consensus.cc:2932] T 450538b8217d40ce945d51e2c171ee36 P 96e5c2fd5d624c2da735e72b387a12d9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6742b9db50c64ea394c2ca38ddee0872 (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { replace: true } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false } } }
I20240502 01:55:26.466696 26690 ts_tablet_manager.cc:1888] T a9c57322de524bce8a114d9a94bd019d P 96e5c2fd5d624c2da735e72b387a12d9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:55:26.467491 26690 log.cc:1196] T a9c57322de524bce8a114d9a94bd019d P 96e5c2fd5d624c2da735e72b387a12d9: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/wal/wals/a9c57322de524bce8a114d9a94bd019d
I20240502 01:55:26.467310 26782 raft_consensus.cc:1050] T 75d71cdcf4d44353acb7d2f18696977b P 96e5c2fd5d624c2da735e72b387a12d9: attempting to promote NON_VOTER 6742b9db50c64ea394c2ca38ddee0872 to VOTER
I20240502 01:55:26.469097 24712 catalog_manager.cc:4830] TS 96e5c2fd5d624c2da735e72b387a12d9 (127.9.176.193:32783): tablet a9c57322de524bce8a114d9a94bd019d (table rebalance_test_table_2 [id=26fb053e91c344cab36e6f0312de1a3c]) successfully deleted
I20240502 01:55:26.470937 26782 consensus_queue.cc:237] T 75d71cdcf4d44353acb7d2f18696977b P 96e5c2fd5d624c2da735e72b387a12d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { replace: true } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false } }
I20240502 01:55:26.475836 25133 consensus_queue.cc:237] T 450538b8217d40ce945d51e2c171ee36 P 672a55e7d4874c5d99f3c5e3d9550f15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false } }
I20240502 01:55:26.478379 25933 raft_consensus.cc:2932] T 32b0189cb19f42efbe7118003e9390f9 P 30921110704f47f88d68a7d9364d42cc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 672a55e7d4874c5d99f3c5e3d9550f15 (127.9.176.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: false } } }
I20240502 01:55:26.478953 24722 catalog_manager.cc:5484] T 32b0189cb19f42efbe7118003e9390f9 P 75bd3f0e8d7c4e8aa92a353301afeb21 reported cstate change: config changed from index 5 to 6, VOTER 672a55e7d4874c5d99f3c5e3d9550f15 (127.9.176.195) evicted. New cstate: current_term: 1 leader_uuid: "96e5c2fd5d624c2da735e72b387a12d9" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: false } } }
I20240502 01:55:26.480679 24868 raft_consensus.cc:1261] T 450538b8217d40ce945d51e2c171ee36 P 96e5c2fd5d624c2da735e72b387a12d9 [term 1 FOLLOWER]: Refusing update from remote peer 672a55e7d4874c5d99f3c5e3d9550f15: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:26.483709 26780 consensus_queue.cc:1035] T 450538b8217d40ce945d51e2c171ee36 P 672a55e7d4874c5d99f3c5e3d9550f15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:26.449301 26196 raft_consensus.cc:2932] T 450538b8217d40ce945d51e2c171ee36 P 6742b9db50c64ea394c2ca38ddee0872 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6742b9db50c64ea394c2ca38ddee0872 (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { replace: true } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false } } }
I20240502 01:55:26.485733 24712 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 32b0189cb19f42efbe7118003e9390f9 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:55:26.456838 26197 raft_consensus.cc:2932] T b6beb3b47fd545d2b6a2d9181f4e5468 P 6742b9db50c64ea394c2ca38ddee0872 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6742b9db50c64ea394c2ca38ddee0872 (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { replace: true } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false } } }
I20240502 01:55:26.488549 26631 raft_consensus.cc:2932] T 450538b8217d40ce945d51e2c171ee36 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 75bd3f0e8d7c4e8aa92a353301afeb21 (127.9.176.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false } } }
I20240502 01:55:26.492878 24712 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 450538b8217d40ce945d51e2c171ee36 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:55:26.495420 24725 catalog_manager.cc:5484] T 450538b8217d40ce945d51e2c171ee36 P 672a55e7d4874c5d99f3c5e3d9550f15 reported cstate change: config changed from index 5 to 6, VOTER 75bd3f0e8d7c4e8aa92a353301afeb21 (127.9.176.194) evicted. New cstate: current_term: 1 leader_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:55:26.497843 26194 raft_consensus.cc:1261] T 75d71cdcf4d44353acb7d2f18696977b P 6742b9db50c64ea394c2ca38ddee0872 [term 1 LEARNER]: Refusing update from remote peer 96e5c2fd5d624c2da735e72b387a12d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:26.498837 25002 raft_consensus.cc:1261] T 75d71cdcf4d44353acb7d2f18696977b P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 1 FOLLOWER]: Refusing update from remote peer 96e5c2fd5d624c2da735e72b387a12d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:26.498749 26634 consensus_queue.cc:1035] T 75d71cdcf4d44353acb7d2f18696977b P 96e5c2fd5d624c2da735e72b387a12d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:26.499369 25133 raft_consensus.cc:1261] T 75d71cdcf4d44353acb7d2f18696977b P 672a55e7d4874c5d99f3c5e3d9550f15 [term 1 FOLLOWER]: Refusing update from remote peer 96e5c2fd5d624c2da735e72b387a12d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:26.499559 26782 consensus_queue.cc:1035] T 75d71cdcf4d44353acb7d2f18696977b P 96e5c2fd5d624c2da735e72b387a12d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:26.500674 26633 consensus_queue.cc:1035] T 75d71cdcf4d44353acb7d2f18696977b P 96e5c2fd5d624c2da735e72b387a12d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:26.501725 24982 tablet_service.cc:1514] Processing DeleteTablet for tablet 450538b8217d40ce945d51e2c171ee36 with delete_type TABLET_DATA_TOMBSTONED (TS 75bd3f0e8d7c4e8aa92a353301afeb21 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57512
I20240502 01:55:26.503894 24869 raft_consensus.cc:2932] T b6beb3b47fd545d2b6a2d9181f4e5468 P 96e5c2fd5d624c2da735e72b387a12d9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6742b9db50c64ea394c2ca38ddee0872 (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { replace: true } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false } } }
I20240502 01:55:26.514874 25134 consensus_queue.cc:237] T b6beb3b47fd545d2b6a2d9181f4e5468 P 672a55e7d4874c5d99f3c5e3d9550f15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false } }
I20240502 01:55:26.519057 24868 raft_consensus.cc:2932] T 450538b8217d40ce945d51e2c171ee36 P 96e5c2fd5d624c2da735e72b387a12d9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 75bd3f0e8d7c4e8aa92a353301afeb21 (127.9.176.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false } } }
I20240502 01:55:26.521016 26193 raft_consensus.cc:1261] T b6beb3b47fd545d2b6a2d9181f4e5468 P 6742b9db50c64ea394c2ca38ddee0872 [term 1 FOLLOWER]: Refusing update from remote peer 672a55e7d4874c5d99f3c5e3d9550f15: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:26.521975 24867 raft_consensus.cc:1261] T b6beb3b47fd545d2b6a2d9181f4e5468 P 96e5c2fd5d624c2da735e72b387a12d9 [term 1 FOLLOWER]: Refusing update from remote peer 672a55e7d4874c5d99f3c5e3d9550f15: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:26.522941 26780 consensus_queue.cc:1035] T b6beb3b47fd545d2b6a2d9181f4e5468 P 672a55e7d4874c5d99f3c5e3d9550f15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:26.524098 25112 tablet_service.cc:1514] Processing DeleteTablet for tablet 32b0189cb19f42efbe7118003e9390f9 with delete_type TABLET_DATA_TOMBSTONED (TS 672a55e7d4874c5d99f3c5e3d9550f15 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:46766
I20240502 01:55:26.524442 26695 tablet_replica.cc:330] T 32b0189cb19f42efbe7118003e9390f9 P 672a55e7d4874c5d99f3c5e3d9550f15: stopping tablet replica
I20240502 01:55:26.524677 26695 raft_consensus.cc:2229] T 32b0189cb19f42efbe7118003e9390f9 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:26.524917 26695 raft_consensus.cc:2258] T 32b0189cb19f42efbe7118003e9390f9 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:26.525254 26195 raft_consensus.cc:1261] T 450538b8217d40ce945d51e2c171ee36 P 6742b9db50c64ea394c2ca38ddee0872 [term 1 FOLLOWER]: Refusing update from remote peer 672a55e7d4874c5d99f3c5e3d9550f15: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:26.526135 26766 consensus_queue.cc:1035] T b6beb3b47fd545d2b6a2d9181f4e5468 P 672a55e7d4874c5d99f3c5e3d9550f15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:26.526368 26631 consensus_queue.cc:1035] T 450538b8217d40ce945d51e2c171ee36 P 672a55e7d4874c5d99f3c5e3d9550f15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:26.528224 26193 raft_consensus.cc:2932] T 450538b8217d40ce945d51e2c171ee36 P 6742b9db50c64ea394c2ca38ddee0872 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 75bd3f0e8d7c4e8aa92a353301afeb21 (127.9.176.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false } } }
I20240502 01:55:26.528731 26695 ts_tablet_manager.cc:1875] T 32b0189cb19f42efbe7118003e9390f9 P 672a55e7d4874c5d99f3c5e3d9550f15: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:26.529423 26780 raft_consensus.cc:2932] T b6beb3b47fd545d2b6a2d9181f4e5468 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 75bd3f0e8d7c4e8aa92a353301afeb21 (127.9.176.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false } } }
I20240502 01:55:26.532311 26195 raft_consensus.cc:2932] T b6beb3b47fd545d2b6a2d9181f4e5468 P 6742b9db50c64ea394c2ca38ddee0872 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 75bd3f0e8d7c4e8aa92a353301afeb21 (127.9.176.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false } } }
I20240502 01:55:26.537176 24867 raft_consensus.cc:2932] T b6beb3b47fd545d2b6a2d9181f4e5468 P 96e5c2fd5d624c2da735e72b387a12d9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 75bd3f0e8d7c4e8aa92a353301afeb21 (127.9.176.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false } } }
I20240502 01:55:26.540289 24712 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b6beb3b47fd545d2b6a2d9181f4e5468 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:55:26.542426 24725 catalog_manager.cc:5484] T b6beb3b47fd545d2b6a2d9181f4e5468 P 672a55e7d4874c5d99f3c5e3d9550f15 reported cstate change: config changed from index 5 to 6, VOTER 75bd3f0e8d7c4e8aa92a353301afeb21 (127.9.176.194) evicted. New cstate: current_term: 1 leader_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:55:26.546517 26695 ts_tablet_manager.cc:1888] T 32b0189cb19f42efbe7118003e9390f9 P 672a55e7d4874c5d99f3c5e3d9550f15: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:55:26.546813 26695 log.cc:1196] T 32b0189cb19f42efbe7118003e9390f9 P 672a55e7d4874c5d99f3c5e3d9550f15: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/wal/wals/32b0189cb19f42efbe7118003e9390f9
I20240502 01:55:26.548620 24980 tablet_service.cc:1514] Processing DeleteTablet for tablet b6beb3b47fd545d2b6a2d9181f4e5468 with delete_type TABLET_DATA_TOMBSTONED (TS 75bd3f0e8d7c4e8aa92a353301afeb21 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57512
I20240502 01:55:26.549219 24712 catalog_manager.cc:4830] TS 672a55e7d4874c5d99f3c5e3d9550f15 (127.9.176.195:46579): tablet 32b0189cb19f42efbe7118003e9390f9 (table rebalance_test_table_2 [id=26fb053e91c344cab36e6f0312de1a3c]) successfully deleted
I20240502 01:55:26.559996 26796 tablet_replica.cc:330] T 450538b8217d40ce945d51e2c171ee36 P 75bd3f0e8d7c4e8aa92a353301afeb21: stopping tablet replica
I20240502 01:55:26.560772 26796 raft_consensus.cc:2229] T 450538b8217d40ce945d51e2c171ee36 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:26.561347 26796 raft_consensus.cc:2258] T 450538b8217d40ce945d51e2c171ee36 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:26.566249 26634 raft_consensus.cc:2932] T 75d71cdcf4d44353acb7d2f18696977b P 96e5c2fd5d624c2da735e72b387a12d9 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6742b9db50c64ea394c2ca38ddee0872 (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { replace: true } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false } } }
I20240502 01:55:26.569806 26194 raft_consensus.cc:2932] T 75d71cdcf4d44353acb7d2f18696977b P 6742b9db50c64ea394c2ca38ddee0872 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6742b9db50c64ea394c2ca38ddee0872 (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { replace: true } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false } } }
I20240502 01:55:26.572899 25133 raft_consensus.cc:2932] T 75d71cdcf4d44353acb7d2f18696977b P 672a55e7d4874c5d99f3c5e3d9550f15 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6742b9db50c64ea394c2ca38ddee0872 (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { replace: true } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false } } }
I20240502 01:55:26.569012 25000 raft_consensus.cc:2932] T 75d71cdcf4d44353acb7d2f18696977b P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6742b9db50c64ea394c2ca38ddee0872 (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { replace: true } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false } } }
I20240502 01:55:26.573823 26796 ts_tablet_manager.cc:1875] T 450538b8217d40ce945d51e2c171ee36 P 75bd3f0e8d7c4e8aa92a353301afeb21: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:26.577786 24724 catalog_manager.cc:5484] T 75d71cdcf4d44353acb7d2f18696977b P 672a55e7d4874c5d99f3c5e3d9550f15 reported cstate change: config changed from index 4 to 5, 6742b9db50c64ea394c2ca38ddee0872 (127.9.176.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "96e5c2fd5d624c2da735e72b387a12d9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { replace: true } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false } } }
I20240502 01:55:26.583056 26796 ts_tablet_manager.cc:1888] T 450538b8217d40ce945d51e2c171ee36 P 75bd3f0e8d7c4e8aa92a353301afeb21: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:55:26.583355 26796 log.cc:1196] T 450538b8217d40ce945d51e2c171ee36 P 75bd3f0e8d7c4e8aa92a353301afeb21: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/wal/wals/450538b8217d40ce945d51e2c171ee36
I20240502 01:55:26.584884 26796 tablet_replica.cc:330] T b6beb3b47fd545d2b6a2d9181f4e5468 P 75bd3f0e8d7c4e8aa92a353301afeb21: stopping tablet replica
I20240502 01:55:26.585106 26796 raft_consensus.cc:2229] T b6beb3b47fd545d2b6a2d9181f4e5468 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:26.585143 24712 catalog_manager.cc:4830] TS 75bd3f0e8d7c4e8aa92a353301afeb21 (127.9.176.194:41625): tablet 450538b8217d40ce945d51e2c171ee36 (table rebalance_test_table_1 [id=4e02aab5efa847ab8735d911dc7a4f5a]) successfully deleted
I20240502 01:55:26.585471 26796 raft_consensus.cc:2258] T b6beb3b47fd545d2b6a2d9181f4e5468 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:26.586922 26796 ts_tablet_manager.cc:1875] T b6beb3b47fd545d2b6a2d9181f4e5468 P 75bd3f0e8d7c4e8aa92a353301afeb21: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:26.587743 24867 consensus_queue.cc:237] T 75d71cdcf4d44353acb7d2f18696977b P 96e5c2fd5d624c2da735e72b387a12d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false } }
I20240502 01:55:26.590555 26194 raft_consensus.cc:1261] T 75d71cdcf4d44353acb7d2f18696977b P 6742b9db50c64ea394c2ca38ddee0872 [term 1 FOLLOWER]: Refusing update from remote peer 96e5c2fd5d624c2da735e72b387a12d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:26.592173 26782 consensus_queue.cc:1035] T 75d71cdcf4d44353acb7d2f18696977b P 96e5c2fd5d624c2da735e72b387a12d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:26.593631 26796 ts_tablet_manager.cc:1888] T b6beb3b47fd545d2b6a2d9181f4e5468 P 75bd3f0e8d7c4e8aa92a353301afeb21: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:55:26.593777 26796 log.cc:1196] T b6beb3b47fd545d2b6a2d9181f4e5468 P 75bd3f0e8d7c4e8aa92a353301afeb21: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/wal/wals/b6beb3b47fd545d2b6a2d9181f4e5468
I20240502 01:55:26.594794 24712 catalog_manager.cc:4830] TS 75bd3f0e8d7c4e8aa92a353301afeb21 (127.9.176.194:41625): tablet b6beb3b47fd545d2b6a2d9181f4e5468 (table rebalance_test_table_1 [id=4e02aab5efa847ab8735d911dc7a4f5a]) successfully deleted
I20240502 01:55:26.596145 26634 raft_consensus.cc:2932] T 75d71cdcf4d44353acb7d2f18696977b P 96e5c2fd5d624c2da735e72b387a12d9 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 75bd3f0e8d7c4e8aa92a353301afeb21 (127.9.176.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false } } }
I20240502 01:55:26.596779 26194 raft_consensus.cc:2932] T 75d71cdcf4d44353acb7d2f18696977b P 6742b9db50c64ea394c2ca38ddee0872 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 75bd3f0e8d7c4e8aa92a353301afeb21 (127.9.176.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false } } }
I20240502 01:55:26.600077 25133 raft_consensus.cc:1261] T 75d71cdcf4d44353acb7d2f18696977b P 672a55e7d4874c5d99f3c5e3d9550f15 [term 1 FOLLOWER]: Refusing update from remote peer 96e5c2fd5d624c2da735e72b387a12d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:26.600801 24712 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 75d71cdcf4d44353acb7d2f18696977b with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:55:26.601315 26782 consensus_queue.cc:1035] T 75d71cdcf4d44353acb7d2f18696977b P 96e5c2fd5d624c2da735e72b387a12d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:26.602531 24725 catalog_manager.cc:5484] T 75d71cdcf4d44353acb7d2f18696977b P 6742b9db50c64ea394c2ca38ddee0872 reported cstate change: config changed from index 5 to 6, VOTER 75bd3f0e8d7c4e8aa92a353301afeb21 (127.9.176.194) evicted. New cstate: current_term: 1 leader_uuid: "96e5c2fd5d624c2da735e72b387a12d9" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false } } }
I20240502 01:55:26.603385 25133 raft_consensus.cc:2932] T 75d71cdcf4d44353acb7d2f18696977b P 672a55e7d4874c5d99f3c5e3d9550f15 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 75bd3f0e8d7c4e8aa92a353301afeb21 (127.9.176.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false } } }
I20240502 01:55:26.606793 24982 tablet_service.cc:1514] Processing DeleteTablet for tablet 75d71cdcf4d44353acb7d2f18696977b with delete_type TABLET_DATA_TOMBSTONED (TS 75bd3f0e8d7c4e8aa92a353301afeb21 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57512
I20240502 01:55:26.607195 26796 tablet_replica.cc:330] T 75d71cdcf4d44353acb7d2f18696977b P 75bd3f0e8d7c4e8aa92a353301afeb21: stopping tablet replica
I20240502 01:55:26.607405 26796 raft_consensus.cc:2229] T 75d71cdcf4d44353acb7d2f18696977b P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:26.607695 26796 raft_consensus.cc:2258] T 75d71cdcf4d44353acb7d2f18696977b P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:26.608743 26796 ts_tablet_manager.cc:1875] T 75d71cdcf4d44353acb7d2f18696977b P 75bd3f0e8d7c4e8aa92a353301afeb21: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:26.615429 26796 ts_tablet_manager.cc:1888] T 75d71cdcf4d44353acb7d2f18696977b P 75bd3f0e8d7c4e8aa92a353301afeb21: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:55:26.615631 26796 log.cc:1196] T 75d71cdcf4d44353acb7d2f18696977b P 75bd3f0e8d7c4e8aa92a353301afeb21: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/wal/wals/75d71cdcf4d44353acb7d2f18696977b
I20240502 01:55:26.616701 24712 catalog_manager.cc:4830] TS 75bd3f0e8d7c4e8aa92a353301afeb21 (127.9.176.194:41625): tablet 75d71cdcf4d44353acb7d2f18696977b (table rebalance_test_table_1 [id=4e02aab5efa847ab8735d911dc7a4f5a]) successfully deleted
I20240502 01:55:26.750021 25914 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:26.799018 25112 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:55:26.799283 24848 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:55:26.800566 24982 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:55:26.803946 26177 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:26.809682 26045 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:27.011075 25914 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:27.029083 26045 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:27.031453 24848 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:55:27.033347 25112 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:55:27.034550 24982 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:55:27.036664 26177 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:27.135862 25134 consensus_queue.cc:237] T 450538b8217d40ce945d51e2c171ee36 P 672a55e7d4874c5d99f3c5e3d9550f15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } attrs { replace: true } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: NON_VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { promote: true } }
I20240502 01:55:27.138073 26194 raft_consensus.cc:1261] T 450538b8217d40ce945d51e2c171ee36 P 6742b9db50c64ea394c2ca38ddee0872 [term 1 FOLLOWER]: Refusing update from remote peer 672a55e7d4874c5d99f3c5e3d9550f15: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:27.138275 24867 raft_consensus.cc:1261] T 450538b8217d40ce945d51e2c171ee36 P 96e5c2fd5d624c2da735e72b387a12d9 [term 1 FOLLOWER]: Refusing update from remote peer 672a55e7d4874c5d99f3c5e3d9550f15: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 01:55:27.138720 25068 consensus_peers.cc:485] T 450538b8217d40ce945d51e2c171ee36 P 672a55e7d4874c5d99f3c5e3d9550f15 -> Peer 75bd3f0e8d7c4e8aa92a353301afeb21 (127.9.176.194:41625): Couldn't send request to peer 75bd3f0e8d7c4e8aa92a353301afeb21. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:27.139214 26766 consensus_queue.cc:1035] T 450538b8217d40ce945d51e2c171ee36 P 672a55e7d4874c5d99f3c5e3d9550f15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:27.139552 26631 consensus_queue.cc:1035] T 450538b8217d40ce945d51e2c171ee36 P 672a55e7d4874c5d99f3c5e3d9550f15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:27.142285 26780 raft_consensus.cc:2932] T 450538b8217d40ce945d51e2c171ee36 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 75bd3f0e8d7c4e8aa92a353301afeb21 (127.9.176.194) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } attrs { replace: true } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: NON_VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { promote: true } } }
I20240502 01:55:27.142973 26194 raft_consensus.cc:2932] T 450538b8217d40ce945d51e2c171ee36 P 6742b9db50c64ea394c2ca38ddee0872 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 75bd3f0e8d7c4e8aa92a353301afeb21 (127.9.176.194) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } attrs { replace: true } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: NON_VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { promote: true } } }
I20240502 01:55:27.146423 24867 raft_consensus.cc:2932] T 450538b8217d40ce945d51e2c171ee36 P 96e5c2fd5d624c2da735e72b387a12d9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 75bd3f0e8d7c4e8aa92a353301afeb21 (127.9.176.194) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } attrs { replace: true } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: NON_VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { promote: true } } }
I20240502 01:55:27.149772 24726 catalog_manager.cc:5484] T 450538b8217d40ce945d51e2c171ee36 P 6742b9db50c64ea394c2ca38ddee0872 reported cstate change: config changed from index 6 to 7, NON_VOTER 75bd3f0e8d7c4e8aa92a353301afeb21 (127.9.176.194) added. New cstate: current_term: 1 leader_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } attrs { replace: true } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: NON_VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { promote: true } } }
I20240502 01:55:27.176527 24867 consensus_queue.cc:237] T 75d71cdcf4d44353acb7d2f18696977b P 96e5c2fd5d624c2da735e72b387a12d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } attrs { replace: true } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: NON_VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { promote: true } }
W20240502 01:55:27.179126 24803 consensus_peers.cc:485] T 75d71cdcf4d44353acb7d2f18696977b P 96e5c2fd5d624c2da735e72b387a12d9 -> Peer 75bd3f0e8d7c4e8aa92a353301afeb21 (127.9.176.194:41625): Couldn't send request to peer 75bd3f0e8d7c4e8aa92a353301afeb21. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:27.179180 26194 raft_consensus.cc:1261] T 75d71cdcf4d44353acb7d2f18696977b P 6742b9db50c64ea394c2ca38ddee0872 [term 1 FOLLOWER]: Refusing update from remote peer 96e5c2fd5d624c2da735e72b387a12d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:27.179168 25133 raft_consensus.cc:1261] T 75d71cdcf4d44353acb7d2f18696977b P 672a55e7d4874c5d99f3c5e3d9550f15 [term 1 FOLLOWER]: Refusing update from remote peer 96e5c2fd5d624c2da735e72b387a12d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:27.179847 26634 consensus_queue.cc:1035] T 75d71cdcf4d44353acb7d2f18696977b P 96e5c2fd5d624c2da735e72b387a12d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:27.180188 26914 consensus_queue.cc:1035] T 75d71cdcf4d44353acb7d2f18696977b P 96e5c2fd5d624c2da735e72b387a12d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:27.182857 26634 raft_consensus.cc:2932] T 75d71cdcf4d44353acb7d2f18696977b P 96e5c2fd5d624c2da735e72b387a12d9 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 75bd3f0e8d7c4e8aa92a353301afeb21 (127.9.176.194) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } attrs { replace: true } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: NON_VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { promote: true } } }
I20240502 01:55:27.183470 26194 raft_consensus.cc:2932] T 75d71cdcf4d44353acb7d2f18696977b P 6742b9db50c64ea394c2ca38ddee0872 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 75bd3f0e8d7c4e8aa92a353301afeb21 (127.9.176.194) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } attrs { replace: true } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: NON_VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { promote: true } } }
I20240502 01:55:27.183503 25133 raft_consensus.cc:2932] T 75d71cdcf4d44353acb7d2f18696977b P 672a55e7d4874c5d99f3c5e3d9550f15 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 75bd3f0e8d7c4e8aa92a353301afeb21 (127.9.176.194) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } attrs { replace: true } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: NON_VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { promote: true } } }
I20240502 01:55:27.189589 24722 catalog_manager.cc:5484] T 75d71cdcf4d44353acb7d2f18696977b P 672a55e7d4874c5d99f3c5e3d9550f15 reported cstate change: config changed from index 6 to 7, NON_VOTER 75bd3f0e8d7c4e8aa92a353301afeb21 (127.9.176.194) added. New cstate: current_term: 1 leader_uuid: "96e5c2fd5d624c2da735e72b387a12d9" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } attrs { replace: true } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: NON_VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { promote: true } } }
I20240502 01:55:27.231252 25133 consensus_queue.cc:237] T ce37ee2e946b4075b78f3c9456939848 P 672a55e7d4874c5d99f3c5e3d9550f15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { replace: true } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: false } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: NON_VOTER last_known_addr { host: "127.9.176.193" port: 32783 } attrs { promote: true } }
I20240502 01:55:27.233779 26066 raft_consensus.cc:1261] T ce37ee2e946b4075b78f3c9456939848 P d490b54f0e954ef59deb6750523f6027 [term 1 FOLLOWER]: Refusing update from remote peer 672a55e7d4874c5d99f3c5e3d9550f15: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 01:55:27.234776 25068 consensus_peers.cc:485] T ce37ee2e946b4075b78f3c9456939848 P 672a55e7d4874c5d99f3c5e3d9550f15 -> Peer 96e5c2fd5d624c2da735e72b387a12d9 (127.9.176.193:32783): Couldn't send request to peer 96e5c2fd5d624c2da735e72b387a12d9. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:27.234809 25000 raft_consensus.cc:1261] T ce37ee2e946b4075b78f3c9456939848 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 1 FOLLOWER]: Refusing update from remote peer 672a55e7d4874c5d99f3c5e3d9550f15: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:27.235632 26780 consensus_queue.cc:1035] T ce37ee2e946b4075b78f3c9456939848 P 672a55e7d4874c5d99f3c5e3d9550f15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:27.236018 26766 consensus_queue.cc:1035] T ce37ee2e946b4075b78f3c9456939848 P 672a55e7d4874c5d99f3c5e3d9550f15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:27.238689 26631 raft_consensus.cc:2932] T ce37ee2e946b4075b78f3c9456939848 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 96e5c2fd5d624c2da735e72b387a12d9 (127.9.176.193) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { replace: true } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: false } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: NON_VOTER last_known_addr { host: "127.9.176.193" port: 32783 } attrs { promote: true } } }
I20240502 01:55:27.239379 25000 raft_consensus.cc:2932] T ce37ee2e946b4075b78f3c9456939848 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 96e5c2fd5d624c2da735e72b387a12d9 (127.9.176.193) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { replace: true } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: false } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: NON_VOTER last_known_addr { host: "127.9.176.193" port: 32783 } attrs { promote: true } } }
I20240502 01:55:27.241499 26066 raft_consensus.cc:2932] T ce37ee2e946b4075b78f3c9456939848 P d490b54f0e954ef59deb6750523f6027 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 96e5c2fd5d624c2da735e72b387a12d9 (127.9.176.193) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { replace: true } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: false } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: NON_VOTER last_known_addr { host: "127.9.176.193" port: 32783 } attrs { promote: true } } }
I20240502 01:55:27.245148 24726 catalog_manager.cc:5484] T ce37ee2e946b4075b78f3c9456939848 P 672a55e7d4874c5d99f3c5e3d9550f15 reported cstate change: config changed from index 6 to 7, NON_VOTER 96e5c2fd5d624c2da735e72b387a12d9 (127.9.176.193) added. New cstate: current_term: 1 leader_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: NON_VOTER last_known_addr { host: "127.9.176.193" port: 32783 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:55:27.272405 24867 consensus_queue.cc:237] T 32b0189cb19f42efbe7118003e9390f9 P 96e5c2fd5d624c2da735e72b387a12d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { replace: true } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: false } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: NON_VOTER last_known_addr { host: "127.9.176.195" port: 46579 } attrs { promote: true } }
I20240502 01:55:27.275032 25933 raft_consensus.cc:1261] T 32b0189cb19f42efbe7118003e9390f9 P 30921110704f47f88d68a7d9364d42cc [term 1 FOLLOWER]: Refusing update from remote peer 96e5c2fd5d624c2da735e72b387a12d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:27.275655 25000 raft_consensus.cc:1261] T 32b0189cb19f42efbe7118003e9390f9 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 1 FOLLOWER]: Refusing update from remote peer 96e5c2fd5d624c2da735e72b387a12d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:27.276062 26914 consensus_queue.cc:1035] T 32b0189cb19f42efbe7118003e9390f9 P 96e5c2fd5d624c2da735e72b387a12d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 01:55:27.277014 24803 consensus_peers.cc:485] T 32b0189cb19f42efbe7118003e9390f9 P 96e5c2fd5d624c2da735e72b387a12d9 -> Peer 672a55e7d4874c5d99f3c5e3d9550f15 (127.9.176.195:46579): Couldn't send request to peer 672a55e7d4874c5d99f3c5e3d9550f15. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:27.277316 26914 consensus_queue.cc:1035] T 32b0189cb19f42efbe7118003e9390f9 P 96e5c2fd5d624c2da735e72b387a12d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:27.278972 26914 raft_consensus.cc:2932] T 32b0189cb19f42efbe7118003e9390f9 P 96e5c2fd5d624c2da735e72b387a12d9 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 672a55e7d4874c5d99f3c5e3d9550f15 (127.9.176.195) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { replace: true } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: false } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: NON_VOTER last_known_addr { host: "127.9.176.195" port: 46579 } attrs { promote: true } } }
I20240502 01:55:27.284121 25933 raft_consensus.cc:2932] T 32b0189cb19f42efbe7118003e9390f9 P 30921110704f47f88d68a7d9364d42cc [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 672a55e7d4874c5d99f3c5e3d9550f15 (127.9.176.195) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { replace: true } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: false } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: NON_VOTER last_known_addr { host: "127.9.176.195" port: 46579 } attrs { promote: true } } }
I20240502 01:55:27.285657 25000 raft_consensus.cc:2932] T 32b0189cb19f42efbe7118003e9390f9 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 672a55e7d4874c5d99f3c5e3d9550f15 (127.9.176.195) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { replace: true } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: false } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: NON_VOTER last_known_addr { host: "127.9.176.195" port: 46579 } attrs { promote: true } } }
I20240502 01:55:27.288864 24726 catalog_manager.cc:5484] T 32b0189cb19f42efbe7118003e9390f9 P 96e5c2fd5d624c2da735e72b387a12d9 reported cstate change: config changed from index 6 to 7, NON_VOTER 672a55e7d4874c5d99f3c5e3d9550f15 (127.9.176.195) added. New cstate: current_term: 1 leader_uuid: "96e5c2fd5d624c2da735e72b387a12d9" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: NON_VOTER last_known_addr { host: "127.9.176.195" port: 46579 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:55:27.303949 25112 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:55:27.316875 25914 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:27.319519 26177 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:27.394171 24848 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:55:27.397444 24982 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:55:27.399545 26045 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:27.511310 25133 consensus_queue.cc:237] T b6beb3b47fd545d2b6a2d9181f4e5468 P 672a55e7d4874c5d99f3c5e3d9550f15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false replace: true } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: true } }
I20240502 01:55:27.513716 24867 raft_consensus.cc:1261] T b6beb3b47fd545d2b6a2d9181f4e5468 P 96e5c2fd5d624c2da735e72b387a12d9 [term 1 FOLLOWER]: Refusing update from remote peer 672a55e7d4874c5d99f3c5e3d9550f15: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:27.514000 26194 raft_consensus.cc:1261] T b6beb3b47fd545d2b6a2d9181f4e5468 P 6742b9db50c64ea394c2ca38ddee0872 [term 1 FOLLOWER]: Refusing update from remote peer 672a55e7d4874c5d99f3c5e3d9550f15: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:27.514612 26780 consensus_queue.cc:1035] T b6beb3b47fd545d2b6a2d9181f4e5468 P 672a55e7d4874c5d99f3c5e3d9550f15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 01:55:27.514976 25069 consensus_peers.cc:485] T b6beb3b47fd545d2b6a2d9181f4e5468 P 672a55e7d4874c5d99f3c5e3d9550f15 -> Peer d490b54f0e954ef59deb6750523f6027 (127.9.176.198:34317): Couldn't send request to peer d490b54f0e954ef59deb6750523f6027. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b6beb3b47fd545d2b6a2d9181f4e5468. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:27.515270 26780 consensus_queue.cc:1035] T b6beb3b47fd545d2b6a2d9181f4e5468 P 672a55e7d4874c5d99f3c5e3d9550f15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:27.517272 26766 raft_consensus.cc:2932] T b6beb3b47fd545d2b6a2d9181f4e5468 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d490b54f0e954ef59deb6750523f6027 (127.9.176.198) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false replace: true } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: true } } }
I20240502 01:55:27.520244 24867 raft_consensus.cc:2932] T b6beb3b47fd545d2b6a2d9181f4e5468 P 96e5c2fd5d624c2da735e72b387a12d9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d490b54f0e954ef59deb6750523f6027 (127.9.176.198) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false replace: true } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: true } } }
I20240502 01:55:27.521677 26194 raft_consensus.cc:2932] T b6beb3b47fd545d2b6a2d9181f4e5468 P 6742b9db50c64ea394c2ca38ddee0872 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d490b54f0e954ef59deb6750523f6027 (127.9.176.198) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false replace: true } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: true } } }
I20240502 01:55:27.526713 24725 catalog_manager.cc:5484] T b6beb3b47fd545d2b6a2d9181f4e5468 P 6742b9db50c64ea394c2ca38ddee0872 reported cstate change: config changed from index 6 to 7, NON_VOTER d490b54f0e954ef59deb6750523f6027 (127.9.176.198) added. New cstate: current_term: 1 leader_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false replace: true } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: true } } }
I20240502 01:55:27.538980 25000 consensus_queue.cc:237] T 08bb5eb2c2814019b6eda6fbbc288245 P 75bd3f0e8d7c4e8aa92a353301afeb21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: false replace: true } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: true } }
I20240502 01:55:27.541496 25933 raft_consensus.cc:1261] T 08bb5eb2c2814019b6eda6fbbc288245 P 30921110704f47f88d68a7d9364d42cc [term 1 FOLLOWER]: Refusing update from remote peer 75bd3f0e8d7c4e8aa92a353301afeb21: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:27.541569 24867 raft_consensus.cc:1261] T 08bb5eb2c2814019b6eda6fbbc288245 P 96e5c2fd5d624c2da735e72b387a12d9 [term 1 FOLLOWER]: Refusing update from remote peer 75bd3f0e8d7c4e8aa92a353301afeb21: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:27.542302 26995 consensus_queue.cc:1035] T 08bb5eb2c2814019b6eda6fbbc288245 P 75bd3f0e8d7c4e8aa92a353301afeb21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:27.542682 26613 consensus_queue.cc:1035] T 08bb5eb2c2814019b6eda6fbbc288245 P 75bd3f0e8d7c4e8aa92a353301afeb21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:27.545408 26613 raft_consensus.cc:2932] T 08bb5eb2c2814019b6eda6fbbc288245 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6742b9db50c64ea394c2ca38ddee0872 (127.9.176.197) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: false replace: true } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: true } } }
I20240502 01:55:27.546087 24867 raft_consensus.cc:2932] T 08bb5eb2c2814019b6eda6fbbc288245 P 96e5c2fd5d624c2da735e72b387a12d9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6742b9db50c64ea394c2ca38ddee0872 (127.9.176.197) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: false replace: true } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: true } } }
I20240502 01:55:27.552191 24726 catalog_manager.cc:5484] T 08bb5eb2c2814019b6eda6fbbc288245 P 75bd3f0e8d7c4e8aa92a353301afeb21 reported cstate change: config changed from index 6 to 7, NON_VOTER 6742b9db50c64ea394c2ca38ddee0872 (127.9.176.197) added. New cstate: current_term: 1 leader_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: false replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 01:55:27.566318 24934 consensus_peers.cc:485] T 08bb5eb2c2814019b6eda6fbbc288245 P 75bd3f0e8d7c4e8aa92a353301afeb21 -> Peer 6742b9db50c64ea394c2ca38ddee0872 (127.9.176.197:43479): Couldn't send request to peer 6742b9db50c64ea394c2ca38ddee0872. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 08bb5eb2c2814019b6eda6fbbc288245. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:27.569048 25933 raft_consensus.cc:2932] T 08bb5eb2c2814019b6eda6fbbc288245 P 30921110704f47f88d68a7d9364d42cc [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6742b9db50c64ea394c2ca38ddee0872 (127.9.176.197) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: false replace: true } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: true } } }
I20240502 01:55:27.570170 26999 ts_tablet_manager.cc:902] T 450538b8217d40ce945d51e2c171ee36 P 75bd3f0e8d7c4e8aa92a353301afeb21: Initiating tablet copy from peer 672a55e7d4874c5d99f3c5e3d9550f15 (127.9.176.195:46579)
I20240502 01:55:27.576565 26999 tablet_copy_client.cc:308] T 450538b8217d40ce945d51e2c171ee36 P 75bd3f0e8d7c4e8aa92a353301afeb21: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.195:46579
I20240502 01:55:27.577605 25154 tablet_copy_service.cc:140] P 672a55e7d4874c5d99f3c5e3d9550f15: Received BeginTabletCopySession request for tablet 450538b8217d40ce945d51e2c171ee36 from peer 75bd3f0e8d7c4e8aa92a353301afeb21 ({username='slave'} at 127.9.176.194:36123)
I20240502 01:55:27.577821 25154 tablet_copy_service.cc:161] P 672a55e7d4874c5d99f3c5e3d9550f15: Beginning new tablet copy session on tablet 450538b8217d40ce945d51e2c171ee36 from peer 75bd3f0e8d7c4e8aa92a353301afeb21 at {username='slave'} at 127.9.176.194:36123: session id = 75bd3f0e8d7c4e8aa92a353301afeb21-450538b8217d40ce945d51e2c171ee36
I20240502 01:55:27.580024 25154 tablet_copy_source_session.cc:205] T 450538b8217d40ce945d51e2c171ee36 P 672a55e7d4874c5d99f3c5e3d9550f15: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:27.581558 26999 ts_tablet_manager.cc:1875] T 450538b8217d40ce945d51e2c171ee36 P 75bd3f0e8d7c4e8aa92a353301afeb21: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 01:55:27.583537 25000 consensus_queue.cc:237] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 75bd3f0e8d7c4e8aa92a353301afeb21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: false replace: true } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: true } }
I20240502 01:55:27.586251 26066 raft_consensus.cc:1261] T 1b76f351e8644d9e8ff27e3f1a963ba2 P d490b54f0e954ef59deb6750523f6027 [term 1 FOLLOWER]: Refusing update from remote peer 75bd3f0e8d7c4e8aa92a353301afeb21: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:27.586681 25133 raft_consensus.cc:1261] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 1 FOLLOWER]: Refusing update from remote peer 75bd3f0e8d7c4e8aa92a353301afeb21: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:27.587002 27000 consensus_queue.cc:1035] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 75bd3f0e8d7c4e8aa92a353301afeb21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:27.587365 26995 consensus_queue.cc:1035] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 75bd3f0e8d7c4e8aa92a353301afeb21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 01:55:27.587693 24934 consensus_peers.cc:485] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 75bd3f0e8d7c4e8aa92a353301afeb21 -> Peer 6742b9db50c64ea394c2ca38ddee0872 (127.9.176.197:43479): Couldn't send request to peer 6742b9db50c64ea394c2ca38ddee0872. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1b76f351e8644d9e8ff27e3f1a963ba2. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:27.590260 26999 ts_tablet_manager.cc:1888] T 450538b8217d40ce945d51e2c171ee36 P 75bd3f0e8d7c4e8aa92a353301afeb21: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240502 01:55:27.590633 26999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 450538b8217d40ce945d51e2c171ee36. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:27.590509 26995 raft_consensus.cc:2932] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6742b9db50c64ea394c2ca38ddee0872 (127.9.176.197) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: false replace: true } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: true } } }
I20240502 01:55:27.591789 25133 raft_consensus.cc:2932] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6742b9db50c64ea394c2ca38ddee0872 (127.9.176.197) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: false replace: true } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: true } } }
I20240502 01:55:27.593750 26066 raft_consensus.cc:2932] T 1b76f351e8644d9e8ff27e3f1a963ba2 P d490b54f0e954ef59deb6750523f6027 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6742b9db50c64ea394c2ca38ddee0872 (127.9.176.197) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: false replace: true } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: true } } }
I20240502 01:55:27.594614 26999 tablet_copy_client.cc:783] T 450538b8217d40ce945d51e2c171ee36 P 75bd3f0e8d7c4e8aa92a353301afeb21: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:27.594966 26999 tablet_copy_client.cc:647] T 450538b8217d40ce945d51e2c171ee36 P 75bd3f0e8d7c4e8aa92a353301afeb21: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:27.597913 24726 catalog_manager.cc:5484] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 75bd3f0e8d7c4e8aa92a353301afeb21 reported cstate change: config changed from index 6 to 7, NON_VOTER 6742b9db50c64ea394c2ca38ddee0872 (127.9.176.197) added. New cstate: current_term: 1 leader_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:55:27.598857 26999 tablet_copy_client.cc:523] T 450538b8217d40ce945d51e2c171ee36 P 75bd3f0e8d7c4e8aa92a353301afeb21: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:27.606400 26999 tablet_bootstrap.cc:492] T 450538b8217d40ce945d51e2c171ee36 P 75bd3f0e8d7c4e8aa92a353301afeb21: Bootstrap starting.
I20240502 01:55:27.643939 26999 tablet_bootstrap.cc:492] T 450538b8217d40ce945d51e2c171ee36 P 75bd3f0e8d7c4e8aa92a353301afeb21: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:27.644361 26999 tablet_bootstrap.cc:492] T 450538b8217d40ce945d51e2c171ee36 P 75bd3f0e8d7c4e8aa92a353301afeb21: Bootstrap complete.
I20240502 01:55:27.644603 26999 ts_tablet_manager.cc:1367] T 450538b8217d40ce945d51e2c171ee36 P 75bd3f0e8d7c4e8aa92a353301afeb21: Time spent bootstrapping tablet: real 0.038s	user 0.033s	sys 0.004s
I20240502 01:55:27.645246 26999 raft_consensus.cc:348] T 450538b8217d40ce945d51e2c171ee36 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } attrs { replace: true } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: NON_VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { promote: true } }
I20240502 01:55:27.645455 26999 raft_consensus.cc:729] T 450538b8217d40ce945d51e2c171ee36 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 75bd3f0e8d7c4e8aa92a353301afeb21, State: Initialized, Role: LEARNER
I20240502 01:55:27.645606 26999 consensus_queue.cc:260] T 450538b8217d40ce945d51e2c171ee36 P 75bd3f0e8d7c4e8aa92a353301afeb21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } attrs { replace: true } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: NON_VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { promote: true } }
I20240502 01:55:27.646059 26999 ts_tablet_manager.cc:1398] T 450538b8217d40ce945d51e2c171ee36 P 75bd3f0e8d7c4e8aa92a353301afeb21: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240502 01:55:27.646883 25154 tablet_copy_service.cc:342] P 672a55e7d4874c5d99f3c5e3d9550f15: Request end of tablet copy session 75bd3f0e8d7c4e8aa92a353301afeb21-450538b8217d40ce945d51e2c171ee36 received from {username='slave'} at 127.9.176.194:36123
I20240502 01:55:27.647035 25154 tablet_copy_service.cc:433] P 672a55e7d4874c5d99f3c5e3d9550f15: ending tablet copy session 75bd3f0e8d7c4e8aa92a353301afeb21-450538b8217d40ce945d51e2c171ee36 on tablet 450538b8217d40ce945d51e2c171ee36 with peer 75bd3f0e8d7c4e8aa92a353301afeb21
I20240502 01:55:27.678822 27029 ts_tablet_manager.cc:902] T ce37ee2e946b4075b78f3c9456939848 P 96e5c2fd5d624c2da735e72b387a12d9: Initiating tablet copy from peer 672a55e7d4874c5d99f3c5e3d9550f15 (127.9.176.195:46579)
I20240502 01:55:27.680771 27029 tablet_copy_client.cc:308] T ce37ee2e946b4075b78f3c9456939848 P 96e5c2fd5d624c2da735e72b387a12d9: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.195:46579
I20240502 01:55:27.681819 25154 tablet_copy_service.cc:140] P 672a55e7d4874c5d99f3c5e3d9550f15: Received BeginTabletCopySession request for tablet ce37ee2e946b4075b78f3c9456939848 from peer 96e5c2fd5d624c2da735e72b387a12d9 ({username='slave'} at 127.9.176.193:42989)
I20240502 01:55:27.682005 25154 tablet_copy_service.cc:161] P 672a55e7d4874c5d99f3c5e3d9550f15: Beginning new tablet copy session on tablet ce37ee2e946b4075b78f3c9456939848 from peer 96e5c2fd5d624c2da735e72b387a12d9 at {username='slave'} at 127.9.176.193:42989: session id = 96e5c2fd5d624c2da735e72b387a12d9-ce37ee2e946b4075b78f3c9456939848
I20240502 01:55:27.684142 25154 tablet_copy_source_session.cc:205] T ce37ee2e946b4075b78f3c9456939848 P 672a55e7d4874c5d99f3c5e3d9550f15: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:27.685503 27029 ts_tablet_manager.cc:1875] T ce37ee2e946b4075b78f3c9456939848 P 96e5c2fd5d624c2da735e72b387a12d9: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 01:55:27.690086 27029 ts_tablet_manager.cc:1888] T ce37ee2e946b4075b78f3c9456939848 P 96e5c2fd5d624c2da735e72b387a12d9: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240502 01:55:27.690374 27029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce37ee2e946b4075b78f3c9456939848. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:27.692909 27029 tablet_copy_client.cc:783] T ce37ee2e946b4075b78f3c9456939848 P 96e5c2fd5d624c2da735e72b387a12d9: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:27.693208 27029 tablet_copy_client.cc:647] T ce37ee2e946b4075b78f3c9456939848 P 96e5c2fd5d624c2da735e72b387a12d9: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:27.695395 27029 tablet_copy_client.cc:523] T ce37ee2e946b4075b78f3c9456939848 P 96e5c2fd5d624c2da735e72b387a12d9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:27.699785 27029 tablet_bootstrap.cc:492] T ce37ee2e946b4075b78f3c9456939848 P 96e5c2fd5d624c2da735e72b387a12d9: Bootstrap starting.
I20240502 01:55:27.724228 26999 ts_tablet_manager.cc:902] T 75d71cdcf4d44353acb7d2f18696977b P 75bd3f0e8d7c4e8aa92a353301afeb21: Initiating tablet copy from peer 96e5c2fd5d624c2da735e72b387a12d9 (127.9.176.193:32783)
I20240502 01:55:27.725533 26999 tablet_copy_client.cc:308] T 75d71cdcf4d44353acb7d2f18696977b P 75bd3f0e8d7c4e8aa92a353301afeb21: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.193:32783
I20240502 01:55:27.726326 24889 tablet_copy_service.cc:140] P 96e5c2fd5d624c2da735e72b387a12d9: Received BeginTabletCopySession request for tablet 75d71cdcf4d44353acb7d2f18696977b from peer 75bd3f0e8d7c4e8aa92a353301afeb21 ({username='slave'} at 127.9.176.194:42767)
I20240502 01:55:27.726550 24889 tablet_copy_service.cc:161] P 96e5c2fd5d624c2da735e72b387a12d9: Beginning new tablet copy session on tablet 75d71cdcf4d44353acb7d2f18696977b from peer 75bd3f0e8d7c4e8aa92a353301afeb21 at {username='slave'} at 127.9.176.194:42767: session id = 75bd3f0e8d7c4e8aa92a353301afeb21-75d71cdcf4d44353acb7d2f18696977b
I20240502 01:55:27.729355 24889 tablet_copy_source_session.cc:205] T 75d71cdcf4d44353acb7d2f18696977b P 96e5c2fd5d624c2da735e72b387a12d9: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:27.730715 26999 ts_tablet_manager.cc:1875] T 75d71cdcf4d44353acb7d2f18696977b P 75bd3f0e8d7c4e8aa92a353301afeb21: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 01:55:27.736753 26999 ts_tablet_manager.cc:1888] T 75d71cdcf4d44353acb7d2f18696977b P 75bd3f0e8d7c4e8aa92a353301afeb21: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240502 01:55:27.737062 26999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75d71cdcf4d44353acb7d2f18696977b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:27.740342 26999 tablet_copy_client.cc:783] T 75d71cdcf4d44353acb7d2f18696977b P 75bd3f0e8d7c4e8aa92a353301afeb21: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:27.740427 27029 tablet_bootstrap.cc:492] T ce37ee2e946b4075b78f3c9456939848 P 96e5c2fd5d624c2da735e72b387a12d9: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:27.740573 26999 tablet_copy_client.cc:647] T 75d71cdcf4d44353acb7d2f18696977b P 75bd3f0e8d7c4e8aa92a353301afeb21: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:27.740962 27029 tablet_bootstrap.cc:492] T ce37ee2e946b4075b78f3c9456939848 P 96e5c2fd5d624c2da735e72b387a12d9: Bootstrap complete.
I20240502 01:55:27.741254 27029 ts_tablet_manager.cc:1367] T ce37ee2e946b4075b78f3c9456939848 P 96e5c2fd5d624c2da735e72b387a12d9: Time spent bootstrapping tablet: real 0.042s	user 0.038s	sys 0.001s
I20240502 01:55:27.742211 27029 raft_consensus.cc:348] T ce37ee2e946b4075b78f3c9456939848 P 96e5c2fd5d624c2da735e72b387a12d9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { replace: true } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: false } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: NON_VOTER last_known_addr { host: "127.9.176.193" port: 32783 } attrs { promote: true } }
I20240502 01:55:27.742463 27029 raft_consensus.cc:729] T ce37ee2e946b4075b78f3c9456939848 P 96e5c2fd5d624c2da735e72b387a12d9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 96e5c2fd5d624c2da735e72b387a12d9, State: Initialized, Role: LEARNER
I20240502 01:55:27.742669 27029 consensus_queue.cc:260] T ce37ee2e946b4075b78f3c9456939848 P 96e5c2fd5d624c2da735e72b387a12d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { replace: true } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: false } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: NON_VOTER last_known_addr { host: "127.9.176.193" port: 32783 } attrs { promote: true } }
I20240502 01:55:27.743809 27029 ts_tablet_manager.cc:1398] T ce37ee2e946b4075b78f3c9456939848 P 96e5c2fd5d624c2da735e72b387a12d9: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.000s
I20240502 01:55:27.743863 26999 tablet_copy_client.cc:523] T 75d71cdcf4d44353acb7d2f18696977b P 75bd3f0e8d7c4e8aa92a353301afeb21: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:27.744627 25154 tablet_copy_service.cc:342] P 672a55e7d4874c5d99f3c5e3d9550f15: Request end of tablet copy session 96e5c2fd5d624c2da735e72b387a12d9-ce37ee2e946b4075b78f3c9456939848 received from {username='slave'} at 127.9.176.193:42989
I20240502 01:55:27.744755 25154 tablet_copy_service.cc:433] P 672a55e7d4874c5d99f3c5e3d9550f15: ending tablet copy session 96e5c2fd5d624c2da735e72b387a12d9-ce37ee2e946b4075b78f3c9456939848 on tablet ce37ee2e946b4075b78f3c9456939848 with peer 96e5c2fd5d624c2da735e72b387a12d9
I20240502 01:55:27.747187 26999 tablet_bootstrap.cc:492] T 75d71cdcf4d44353acb7d2f18696977b P 75bd3f0e8d7c4e8aa92a353301afeb21: Bootstrap starting.
I20240502 01:55:27.788072 26999 tablet_bootstrap.cc:492] T 75d71cdcf4d44353acb7d2f18696977b P 75bd3f0e8d7c4e8aa92a353301afeb21: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:27.788516 26999 tablet_bootstrap.cc:492] T 75d71cdcf4d44353acb7d2f18696977b P 75bd3f0e8d7c4e8aa92a353301afeb21: Bootstrap complete.
I20240502 01:55:27.788772 26999 ts_tablet_manager.cc:1367] T 75d71cdcf4d44353acb7d2f18696977b P 75bd3f0e8d7c4e8aa92a353301afeb21: Time spent bootstrapping tablet: real 0.042s	user 0.034s	sys 0.008s
I20240502 01:55:27.789474 26999 raft_consensus.cc:348] T 75d71cdcf4d44353acb7d2f18696977b P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } attrs { replace: true } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: NON_VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { promote: true } }
I20240502 01:55:27.789673 26999 raft_consensus.cc:729] T 75d71cdcf4d44353acb7d2f18696977b P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 75bd3f0e8d7c4e8aa92a353301afeb21, State: Initialized, Role: LEARNER
I20240502 01:55:27.789906 26999 consensus_queue.cc:260] T 75d71cdcf4d44353acb7d2f18696977b P 75bd3f0e8d7c4e8aa92a353301afeb21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } attrs { replace: true } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: NON_VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { promote: true } }
I20240502 01:55:27.790416 26999 ts_tablet_manager.cc:1398] T 75d71cdcf4d44353acb7d2f18696977b P 75bd3f0e8d7c4e8aa92a353301afeb21: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 01:55:27.791317 24889 tablet_copy_service.cc:342] P 96e5c2fd5d624c2da735e72b387a12d9: Request end of tablet copy session 75bd3f0e8d7c4e8aa92a353301afeb21-75d71cdcf4d44353acb7d2f18696977b received from {username='slave'} at 127.9.176.194:42767
I20240502 01:55:27.791437 24889 tablet_copy_service.cc:433] P 96e5c2fd5d624c2da735e72b387a12d9: ending tablet copy session 75bd3f0e8d7c4e8aa92a353301afeb21-75d71cdcf4d44353acb7d2f18696977b on tablet 75d71cdcf4d44353acb7d2f18696977b with peer 75bd3f0e8d7c4e8aa92a353301afeb21
I20240502 01:55:27.889377 27075 ts_tablet_manager.cc:902] T 32b0189cb19f42efbe7118003e9390f9 P 672a55e7d4874c5d99f3c5e3d9550f15: Initiating tablet copy from peer 96e5c2fd5d624c2da735e72b387a12d9 (127.9.176.193:32783)
I20240502 01:55:27.891057 27075 tablet_copy_client.cc:308] T 32b0189cb19f42efbe7118003e9390f9 P 672a55e7d4874c5d99f3c5e3d9550f15: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.193:32783
I20240502 01:55:27.892100 24889 tablet_copy_service.cc:140] P 96e5c2fd5d624c2da735e72b387a12d9: Received BeginTabletCopySession request for tablet 32b0189cb19f42efbe7118003e9390f9 from peer 672a55e7d4874c5d99f3c5e3d9550f15 ({username='slave'} at 127.9.176.195:52571)
I20240502 01:55:27.892283 24889 tablet_copy_service.cc:161] P 96e5c2fd5d624c2da735e72b387a12d9: Beginning new tablet copy session on tablet 32b0189cb19f42efbe7118003e9390f9 from peer 672a55e7d4874c5d99f3c5e3d9550f15 at {username='slave'} at 127.9.176.195:52571: session id = 672a55e7d4874c5d99f3c5e3d9550f15-32b0189cb19f42efbe7118003e9390f9
I20240502 01:55:27.894774 24889 tablet_copy_source_session.cc:205] T 32b0189cb19f42efbe7118003e9390f9 P 96e5c2fd5d624c2da735e72b387a12d9: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:27.896307 27075 ts_tablet_manager.cc:1875] T 32b0189cb19f42efbe7118003e9390f9 P 672a55e7d4874c5d99f3c5e3d9550f15: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 01:55:27.901122 27075 ts_tablet_manager.cc:1888] T 32b0189cb19f42efbe7118003e9390f9 P 672a55e7d4874c5d99f3c5e3d9550f15: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240502 01:55:27.901432 27075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32b0189cb19f42efbe7118003e9390f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:27.904016 27075 tablet_copy_client.cc:783] T 32b0189cb19f42efbe7118003e9390f9 P 672a55e7d4874c5d99f3c5e3d9550f15: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:27.904337 27075 tablet_copy_client.cc:647] T 32b0189cb19f42efbe7118003e9390f9 P 672a55e7d4874c5d99f3c5e3d9550f15: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:27.906540 27075 tablet_copy_client.cc:523] T 32b0189cb19f42efbe7118003e9390f9 P 672a55e7d4874c5d99f3c5e3d9550f15: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:27.909689 27075 tablet_bootstrap.cc:492] T 32b0189cb19f42efbe7118003e9390f9 P 672a55e7d4874c5d99f3c5e3d9550f15: Bootstrap starting.
I20240502 01:55:27.940951 27075 tablet_bootstrap.cc:492] T 32b0189cb19f42efbe7118003e9390f9 P 672a55e7d4874c5d99f3c5e3d9550f15: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:27.941466 27075 tablet_bootstrap.cc:492] T 32b0189cb19f42efbe7118003e9390f9 P 672a55e7d4874c5d99f3c5e3d9550f15: Bootstrap complete.
I20240502 01:55:27.941741 27075 ts_tablet_manager.cc:1367] T 32b0189cb19f42efbe7118003e9390f9 P 672a55e7d4874c5d99f3c5e3d9550f15: Time spent bootstrapping tablet: real 0.032s	user 0.028s	sys 0.004s
I20240502 01:55:27.942485 27075 raft_consensus.cc:348] T 32b0189cb19f42efbe7118003e9390f9 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { replace: true } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: false } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: NON_VOTER last_known_addr { host: "127.9.176.195" port: 46579 } attrs { promote: true } }
I20240502 01:55:27.942739 27075 raft_consensus.cc:729] T 32b0189cb19f42efbe7118003e9390f9 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 672a55e7d4874c5d99f3c5e3d9550f15, State: Initialized, Role: LEARNER
I20240502 01:55:27.942945 27075 consensus_queue.cc:260] T 32b0189cb19f42efbe7118003e9390f9 P 672a55e7d4874c5d99f3c5e3d9550f15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { replace: true } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: false } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: NON_VOTER last_known_addr { host: "127.9.176.195" port: 46579 } attrs { promote: true } }
I20240502 01:55:27.943533 27075 ts_tablet_manager.cc:1398] T 32b0189cb19f42efbe7118003e9390f9 P 672a55e7d4874c5d99f3c5e3d9550f15: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 01:55:27.944520 24889 tablet_copy_service.cc:342] P 96e5c2fd5d624c2da735e72b387a12d9: Request end of tablet copy session 672a55e7d4874c5d99f3c5e3d9550f15-32b0189cb19f42efbe7118003e9390f9 received from {username='slave'} at 127.9.176.195:52571
I20240502 01:55:27.944667 24889 tablet_copy_service.cc:433] P 96e5c2fd5d624c2da735e72b387a12d9: ending tablet copy session 672a55e7d4874c5d99f3c5e3d9550f15-32b0189cb19f42efbe7118003e9390f9 on tablet 32b0189cb19f42efbe7118003e9390f9 with peer 672a55e7d4874c5d99f3c5e3d9550f15
I20240502 01:55:27.961805 27081 ts_tablet_manager.cc:902] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 6742b9db50c64ea394c2ca38ddee0872: Initiating tablet copy from peer 75bd3f0e8d7c4e8aa92a353301afeb21 (127.9.176.194:41625)
I20240502 01:55:27.963351 27081 tablet_copy_client.cc:308] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 6742b9db50c64ea394c2ca38ddee0872: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.194:41625
I20240502 01:55:27.970364 25022 tablet_copy_service.cc:140] P 75bd3f0e8d7c4e8aa92a353301afeb21: Received BeginTabletCopySession request for tablet 1b76f351e8644d9e8ff27e3f1a963ba2 from peer 6742b9db50c64ea394c2ca38ddee0872 ({username='slave'} at 127.9.176.197:38439)
I20240502 01:55:27.970547 25022 tablet_copy_service.cc:161] P 75bd3f0e8d7c4e8aa92a353301afeb21: Beginning new tablet copy session on tablet 1b76f351e8644d9e8ff27e3f1a963ba2 from peer 6742b9db50c64ea394c2ca38ddee0872 at {username='slave'} at 127.9.176.197:38439: session id = 6742b9db50c64ea394c2ca38ddee0872-1b76f351e8644d9e8ff27e3f1a963ba2
I20240502 01:55:27.972712 25022 tablet_copy_source_session.cc:205] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 75bd3f0e8d7c4e8aa92a353301afeb21: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:27.974004 27081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b76f351e8644d9e8ff27e3f1a963ba2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:27.979116 27081 tablet_copy_client.cc:783] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 6742b9db50c64ea394c2ca38ddee0872: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:27.979422 27081 tablet_copy_client.cc:647] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 6742b9db50c64ea394c2ca38ddee0872: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:27.981724 27081 tablet_copy_client.cc:523] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 6742b9db50c64ea394c2ca38ddee0872: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:27.984952 27081 tablet_bootstrap.cc:492] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 6742b9db50c64ea394c2ca38ddee0872: Bootstrap starting.
I20240502 01:55:28.028128 27081 tablet_bootstrap.cc:492] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 6742b9db50c64ea394c2ca38ddee0872: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:28.028553 27081 tablet_bootstrap.cc:492] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 6742b9db50c64ea394c2ca38ddee0872: Bootstrap complete.
I20240502 01:55:28.028800 27081 ts_tablet_manager.cc:1367] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 6742b9db50c64ea394c2ca38ddee0872: Time spent bootstrapping tablet: real 0.044s	user 0.035s	sys 0.008s
I20240502 01:55:28.029457 27081 raft_consensus.cc:348] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 6742b9db50c64ea394c2ca38ddee0872 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: false replace: true } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: true } }
I20240502 01:55:28.029668 27081 raft_consensus.cc:729] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 6742b9db50c64ea394c2ca38ddee0872 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6742b9db50c64ea394c2ca38ddee0872, State: Initialized, Role: LEARNER
I20240502 01:55:28.029819 27081 consensus_queue.cc:260] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 6742b9db50c64ea394c2ca38ddee0872 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: false replace: true } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: true } }
I20240502 01:55:28.030500 27081 ts_tablet_manager.cc:1398] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 6742b9db50c64ea394c2ca38ddee0872: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.001s
I20240502 01:55:28.031435 25022 tablet_copy_service.cc:342] P 75bd3f0e8d7c4e8aa92a353301afeb21: Request end of tablet copy session 6742b9db50c64ea394c2ca38ddee0872-1b76f351e8644d9e8ff27e3f1a963ba2 received from {username='slave'} at 127.9.176.197:38439
I20240502 01:55:28.031594 25022 tablet_copy_service.cc:433] P 75bd3f0e8d7c4e8aa92a353301afeb21: ending tablet copy session 6742b9db50c64ea394c2ca38ddee0872-1b76f351e8644d9e8ff27e3f1a963ba2 on tablet 1b76f351e8644d9e8ff27e3f1a963ba2 with peer 6742b9db50c64ea394c2ca38ddee0872
I20240502 01:55:28.106706 25000 raft_consensus.cc:1203] T 450538b8217d40ce945d51e2c171ee36 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240502 01:55:28.108559 25000 raft_consensus.cc:1203] T 75d71cdcf4d44353acb7d2f18696977b P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240502 01:55:28.121876 27130 ts_tablet_manager.cc:902] T b6beb3b47fd545d2b6a2d9181f4e5468 P d490b54f0e954ef59deb6750523f6027: Initiating tablet copy from peer 672a55e7d4874c5d99f3c5e3d9550f15 (127.9.176.195:46579)
I20240502 01:55:28.123080 27130 tablet_copy_client.cc:308] T b6beb3b47fd545d2b6a2d9181f4e5468 P d490b54f0e954ef59deb6750523f6027: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.195:46579
I20240502 01:55:28.123831 25154 tablet_copy_service.cc:140] P 672a55e7d4874c5d99f3c5e3d9550f15: Received BeginTabletCopySession request for tablet b6beb3b47fd545d2b6a2d9181f4e5468 from peer d490b54f0e954ef59deb6750523f6027 ({username='slave'} at 127.9.176.198:40081)
I20240502 01:55:28.123986 25154 tablet_copy_service.cc:161] P 672a55e7d4874c5d99f3c5e3d9550f15: Beginning new tablet copy session on tablet b6beb3b47fd545d2b6a2d9181f4e5468 from peer d490b54f0e954ef59deb6750523f6027 at {username='slave'} at 127.9.176.198:40081: session id = d490b54f0e954ef59deb6750523f6027-b6beb3b47fd545d2b6a2d9181f4e5468
I20240502 01:55:28.125943 25154 tablet_copy_source_session.cc:205] T b6beb3b47fd545d2b6a2d9181f4e5468 P 672a55e7d4874c5d99f3c5e3d9550f15: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:28.127108 27130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6beb3b47fd545d2b6a2d9181f4e5468. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:28.131394 27130 tablet_copy_client.cc:783] T b6beb3b47fd545d2b6a2d9181f4e5468 P d490b54f0e954ef59deb6750523f6027: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:28.131687 27130 tablet_copy_client.cc:647] T b6beb3b47fd545d2b6a2d9181f4e5468 P d490b54f0e954ef59deb6750523f6027: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:28.133314 27130 tablet_copy_client.cc:523] T b6beb3b47fd545d2b6a2d9181f4e5468 P d490b54f0e954ef59deb6750523f6027: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:28.136044 27130 tablet_bootstrap.cc:492] T b6beb3b47fd545d2b6a2d9181f4e5468 P d490b54f0e954ef59deb6750523f6027: Bootstrap starting.
I20240502 01:55:28.152597 27081 ts_tablet_manager.cc:902] T 08bb5eb2c2814019b6eda6fbbc288245 P 6742b9db50c64ea394c2ca38ddee0872: Initiating tablet copy from peer 75bd3f0e8d7c4e8aa92a353301afeb21 (127.9.176.194:41625)
I20240502 01:55:28.154117 27081 tablet_copy_client.cc:308] T 08bb5eb2c2814019b6eda6fbbc288245 P 6742b9db50c64ea394c2ca38ddee0872: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.194:41625
I20240502 01:55:28.154870 25022 tablet_copy_service.cc:140] P 75bd3f0e8d7c4e8aa92a353301afeb21: Received BeginTabletCopySession request for tablet 08bb5eb2c2814019b6eda6fbbc288245 from peer 6742b9db50c64ea394c2ca38ddee0872 ({username='slave'} at 127.9.176.197:38439)
I20240502 01:55:28.155064 25022 tablet_copy_service.cc:161] P 75bd3f0e8d7c4e8aa92a353301afeb21: Beginning new tablet copy session on tablet 08bb5eb2c2814019b6eda6fbbc288245 from peer 6742b9db50c64ea394c2ca38ddee0872 at {username='slave'} at 127.9.176.197:38439: session id = 6742b9db50c64ea394c2ca38ddee0872-08bb5eb2c2814019b6eda6fbbc288245
I20240502 01:55:28.157092 25022 tablet_copy_source_session.cc:205] T 08bb5eb2c2814019b6eda6fbbc288245 P 75bd3f0e8d7c4e8aa92a353301afeb21: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:28.158288 27081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08bb5eb2c2814019b6eda6fbbc288245. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:28.164558 27081 tablet_copy_client.cc:783] T 08bb5eb2c2814019b6eda6fbbc288245 P 6742b9db50c64ea394c2ca38ddee0872: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:28.164793 27081 tablet_copy_client.cc:647] T 08bb5eb2c2814019b6eda6fbbc288245 P 6742b9db50c64ea394c2ca38ddee0872: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:28.166476 27081 tablet_copy_client.cc:523] T 08bb5eb2c2814019b6eda6fbbc288245 P 6742b9db50c64ea394c2ca38ddee0872: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:28.170014 27081 tablet_bootstrap.cc:492] T 08bb5eb2c2814019b6eda6fbbc288245 P 6742b9db50c64ea394c2ca38ddee0872: Bootstrap starting.
I20240502 01:55:28.177537 27130 tablet_bootstrap.cc:492] T b6beb3b47fd545d2b6a2d9181f4e5468 P d490b54f0e954ef59deb6750523f6027: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:28.178045 27130 tablet_bootstrap.cc:492] T b6beb3b47fd545d2b6a2d9181f4e5468 P d490b54f0e954ef59deb6750523f6027: Bootstrap complete.
I20240502 01:55:28.178380 27130 ts_tablet_manager.cc:1367] T b6beb3b47fd545d2b6a2d9181f4e5468 P d490b54f0e954ef59deb6750523f6027: Time spent bootstrapping tablet: real 0.042s	user 0.035s	sys 0.004s
I20240502 01:55:28.179217 27130 raft_consensus.cc:348] T b6beb3b47fd545d2b6a2d9181f4e5468 P d490b54f0e954ef59deb6750523f6027 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false replace: true } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: true } }
I20240502 01:55:28.179515 27130 raft_consensus.cc:729] T b6beb3b47fd545d2b6a2d9181f4e5468 P d490b54f0e954ef59deb6750523f6027 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d490b54f0e954ef59deb6750523f6027, State: Initialized, Role: LEARNER
I20240502 01:55:28.179791 27130 consensus_queue.cc:260] T b6beb3b47fd545d2b6a2d9181f4e5468 P d490b54f0e954ef59deb6750523f6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false replace: true } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: NON_VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: true } }
I20240502 01:55:28.180794 27130 ts_tablet_manager.cc:1398] T b6beb3b47fd545d2b6a2d9181f4e5468 P d490b54f0e954ef59deb6750523f6027: Time spent starting tablet: real 0.002s	user 0.005s	sys 0.000s
I20240502 01:55:28.181784 25154 tablet_copy_service.cc:342] P 672a55e7d4874c5d99f3c5e3d9550f15: Request end of tablet copy session d490b54f0e954ef59deb6750523f6027-b6beb3b47fd545d2b6a2d9181f4e5468 received from {username='slave'} at 127.9.176.198:40081
I20240502 01:55:28.181960 25154 tablet_copy_service.cc:433] P 672a55e7d4874c5d99f3c5e3d9550f15: ending tablet copy session d490b54f0e954ef59deb6750523f6027-b6beb3b47fd545d2b6a2d9181f4e5468 on tablet b6beb3b47fd545d2b6a2d9181f4e5468 with peer d490b54f0e954ef59deb6750523f6027
I20240502 01:55:28.208696 27081 tablet_bootstrap.cc:492] T 08bb5eb2c2814019b6eda6fbbc288245 P 6742b9db50c64ea394c2ca38ddee0872: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:28.209228 27081 tablet_bootstrap.cc:492] T 08bb5eb2c2814019b6eda6fbbc288245 P 6742b9db50c64ea394c2ca38ddee0872: Bootstrap complete.
I20240502 01:55:28.209513 27081 ts_tablet_manager.cc:1367] T 08bb5eb2c2814019b6eda6fbbc288245 P 6742b9db50c64ea394c2ca38ddee0872: Time spent bootstrapping tablet: real 0.040s	user 0.036s	sys 0.004s
I20240502 01:55:28.210228 27081 raft_consensus.cc:348] T 08bb5eb2c2814019b6eda6fbbc288245 P 6742b9db50c64ea394c2ca38ddee0872 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: false replace: true } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: true } }
I20240502 01:55:28.210450 27081 raft_consensus.cc:729] T 08bb5eb2c2814019b6eda6fbbc288245 P 6742b9db50c64ea394c2ca38ddee0872 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6742b9db50c64ea394c2ca38ddee0872, State: Initialized, Role: LEARNER
I20240502 01:55:28.210687 27081 consensus_queue.cc:260] T 08bb5eb2c2814019b6eda6fbbc288245 P 6742b9db50c64ea394c2ca38ddee0872 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: false replace: true } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: true } }
I20240502 01:55:28.211184 27081 ts_tablet_manager.cc:1398] T 08bb5eb2c2814019b6eda6fbbc288245 P 6742b9db50c64ea394c2ca38ddee0872: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 01:55:28.212061 25022 tablet_copy_service.cc:342] P 75bd3f0e8d7c4e8aa92a353301afeb21: Request end of tablet copy session 6742b9db50c64ea394c2ca38ddee0872-08bb5eb2c2814019b6eda6fbbc288245 received from {username='slave'} at 127.9.176.197:38439
I20240502 01:55:28.212234 25022 tablet_copy_service.cc:433] P 75bd3f0e8d7c4e8aa92a353301afeb21: ending tablet copy session 6742b9db50c64ea394c2ca38ddee0872-08bb5eb2c2814019b6eda6fbbc288245 on tablet 08bb5eb2c2814019b6eda6fbbc288245 with peer 6742b9db50c64ea394c2ca38ddee0872
I20240502 01:55:28.272167 25133 raft_consensus.cc:1203] T 32b0189cb19f42efbe7118003e9390f9 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240502 01:55:28.290272 24867 raft_consensus.cc:1203] T ce37ee2e946b4075b78f3c9456939848 P 96e5c2fd5d624c2da735e72b387a12d9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240502 01:55:28.508949 26195 raft_consensus.cc:1203] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 6742b9db50c64ea394c2ca38ddee0872 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240502 01:55:28.584765 27128 raft_consensus.cc:1050] T 75d71cdcf4d44353acb7d2f18696977b P 96e5c2fd5d624c2da735e72b387a12d9: attempting to promote NON_VOTER 75bd3f0e8d7c4e8aa92a353301afeb21 to VOTER
I20240502 01:55:28.585523 27128 consensus_queue.cc:237] T 75d71cdcf4d44353acb7d2f18696977b P 96e5c2fd5d624c2da735e72b387a12d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } attrs { replace: true } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { promote: false } }
I20240502 01:55:28.587953 25000 raft_consensus.cc:1261] T 75d71cdcf4d44353acb7d2f18696977b P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 1 LEARNER]: Refusing update from remote peer 96e5c2fd5d624c2da735e72b387a12d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:28.587957 26195 raft_consensus.cc:1261] T 75d71cdcf4d44353acb7d2f18696977b P 6742b9db50c64ea394c2ca38ddee0872 [term 1 FOLLOWER]: Refusing update from remote peer 96e5c2fd5d624c2da735e72b387a12d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:28.588330 25133 raft_consensus.cc:1261] T 75d71cdcf4d44353acb7d2f18696977b P 672a55e7d4874c5d99f3c5e3d9550f15 [term 1 FOLLOWER]: Refusing update from remote peer 96e5c2fd5d624c2da735e72b387a12d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:28.588675 27216 consensus_queue.cc:1035] T 75d71cdcf4d44353acb7d2f18696977b P 96e5c2fd5d624c2da735e72b387a12d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:28.589157 27128 consensus_queue.cc:1035] T 75d71cdcf4d44353acb7d2f18696977b P 96e5c2fd5d624c2da735e72b387a12d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:28.589495 26914 consensus_queue.cc:1035] T 75d71cdcf4d44353acb7d2f18696977b P 96e5c2fd5d624c2da735e72b387a12d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:28.592988 27128 raft_consensus.cc:2932] T 75d71cdcf4d44353acb7d2f18696977b P 96e5c2fd5d624c2da735e72b387a12d9 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 75bd3f0e8d7c4e8aa92a353301afeb21 (127.9.176.194) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } attrs { replace: true } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { promote: false } } }
I20240502 01:55:28.593854 25133 raft_consensus.cc:2932] T 75d71cdcf4d44353acb7d2f18696977b P 672a55e7d4874c5d99f3c5e3d9550f15 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 75bd3f0e8d7c4e8aa92a353301afeb21 (127.9.176.194) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } attrs { replace: true } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { promote: false } } }
I20240502 01:55:28.597965 25000 raft_consensus.cc:2932] T 75d71cdcf4d44353acb7d2f18696977b P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 75bd3f0e8d7c4e8aa92a353301afeb21 (127.9.176.194) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } attrs { replace: true } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { promote: false } } }
I20240502 01:55:28.598452 24725 catalog_manager.cc:5484] T 75d71cdcf4d44353acb7d2f18696977b P 96e5c2fd5d624c2da735e72b387a12d9 reported cstate change: config changed from index 7 to 8, 75bd3f0e8d7c4e8aa92a353301afeb21 (127.9.176.194) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "96e5c2fd5d624c2da735e72b387a12d9" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:55:28.609925 26195 raft_consensus.cc:2932] T 75d71cdcf4d44353acb7d2f18696977b P 6742b9db50c64ea394c2ca38ddee0872 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 75bd3f0e8d7c4e8aa92a353301afeb21 (127.9.176.194) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } attrs { replace: true } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { promote: false } } }
I20240502 01:55:28.612604 24867 consensus_queue.cc:237] T 75d71cdcf4d44353acb7d2f18696977b P 96e5c2fd5d624c2da735e72b387a12d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { promote: false } }
I20240502 01:55:28.614810 26195 raft_consensus.cc:1261] T 75d71cdcf4d44353acb7d2f18696977b P 6742b9db50c64ea394c2ca38ddee0872 [term 1 FOLLOWER]: Refusing update from remote peer 96e5c2fd5d624c2da735e72b387a12d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:28.614794 25000 raft_consensus.cc:1261] T 75d71cdcf4d44353acb7d2f18696977b P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 1 FOLLOWER]: Refusing update from remote peer 96e5c2fd5d624c2da735e72b387a12d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:28.615504 26914 consensus_queue.cc:1035] T 75d71cdcf4d44353acb7d2f18696977b P 96e5c2fd5d624c2da735e72b387a12d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:28.615847 27216 consensus_queue.cc:1035] T 75d71cdcf4d44353acb7d2f18696977b P 96e5c2fd5d624c2da735e72b387a12d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:28.618252 27129 raft_consensus.cc:2932] T 75d71cdcf4d44353acb7d2f18696977b P 96e5c2fd5d624c2da735e72b387a12d9 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 672a55e7d4874c5d99f3c5e3d9550f15 (127.9.176.195) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { promote: false } } }
I20240502 01:55:28.618953 26195 raft_consensus.cc:2932] T 75d71cdcf4d44353acb7d2f18696977b P 6742b9db50c64ea394c2ca38ddee0872 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 672a55e7d4874c5d99f3c5e3d9550f15 (127.9.176.195) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { promote: false } } }
I20240502 01:55:28.619036 25000 raft_consensus.cc:2932] T 75d71cdcf4d44353acb7d2f18696977b P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 672a55e7d4874c5d99f3c5e3d9550f15 (127.9.176.195) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { promote: false } } }
I20240502 01:55:28.622767 24712 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 75d71cdcf4d44353acb7d2f18696977b with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:55:28.623607 24724 catalog_manager.cc:5484] T 75d71cdcf4d44353acb7d2f18696977b P 6742b9db50c64ea394c2ca38ddee0872 reported cstate change: config changed from index 8 to 9, VOTER 672a55e7d4874c5d99f3c5e3d9550f15 (127.9.176.195) evicted. New cstate: current_term: 1 leader_uuid: "96e5c2fd5d624c2da735e72b387a12d9" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { promote: false } } }
I20240502 01:55:28.627310 25112 tablet_service.cc:1514] Processing DeleteTablet for tablet 75d71cdcf4d44353acb7d2f18696977b with delete_type TABLET_DATA_TOMBSTONED (TS 672a55e7d4874c5d99f3c5e3d9550f15 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:46766
I20240502 01:55:28.628113 27224 tablet_replica.cc:330] T 75d71cdcf4d44353acb7d2f18696977b P 672a55e7d4874c5d99f3c5e3d9550f15: stopping tablet replica
I20240502 01:55:28.628353 27224 raft_consensus.cc:2229] T 75d71cdcf4d44353acb7d2f18696977b P 672a55e7d4874c5d99f3c5e3d9550f15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:28.628610 27224 raft_consensus.cc:2258] T 75d71cdcf4d44353acb7d2f18696977b P 672a55e7d4874c5d99f3c5e3d9550f15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:28.629565 27224 ts_tablet_manager.cc:1875] T 75d71cdcf4d44353acb7d2f18696977b P 672a55e7d4874c5d99f3c5e3d9550f15: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:28.634073 27224 ts_tablet_manager.cc:1888] T 75d71cdcf4d44353acb7d2f18696977b P 672a55e7d4874c5d99f3c5e3d9550f15: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 01:55:28.634225 27224 log.cc:1196] T 75d71cdcf4d44353acb7d2f18696977b P 672a55e7d4874c5d99f3c5e3d9550f15: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/wal/wals/75d71cdcf4d44353acb7d2f18696977b
I20240502 01:55:28.635084 24712 catalog_manager.cc:4830] TS 672a55e7d4874c5d99f3c5e3d9550f15 (127.9.176.195:46579): tablet 75d71cdcf4d44353acb7d2f18696977b (table rebalance_test_table_1 [id=4e02aab5efa847ab8735d911dc7a4f5a]) successfully deleted
I20240502 01:55:28.642392 26195 raft_consensus.cc:1203] T 08bb5eb2c2814019b6eda6fbbc288245 P 6742b9db50c64ea394c2ca38ddee0872 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240502 01:55:28.652709 26066 raft_consensus.cc:1203] T b6beb3b47fd545d2b6a2d9181f4e5468 P d490b54f0e954ef59deb6750523f6027 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240502 01:55:28.697010 27225 raft_consensus.cc:1050] T 450538b8217d40ce945d51e2c171ee36 P 672a55e7d4874c5d99f3c5e3d9550f15: attempting to promote NON_VOTER 75bd3f0e8d7c4e8aa92a353301afeb21 to VOTER
I20240502 01:55:28.697741 27225 consensus_queue.cc:237] T 450538b8217d40ce945d51e2c171ee36 P 672a55e7d4874c5d99f3c5e3d9550f15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } attrs { replace: true } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { promote: false } }
I20240502 01:55:28.700201 25000 raft_consensus.cc:1261] T 450538b8217d40ce945d51e2c171ee36 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 1 LEARNER]: Refusing update from remote peer 672a55e7d4874c5d99f3c5e3d9550f15: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:28.700240 26195 raft_consensus.cc:1261] T 450538b8217d40ce945d51e2c171ee36 P 6742b9db50c64ea394c2ca38ddee0872 [term 1 FOLLOWER]: Refusing update from remote peer 672a55e7d4874c5d99f3c5e3d9550f15: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:28.700304 24867 raft_consensus.cc:1261] T 450538b8217d40ce945d51e2c171ee36 P 96e5c2fd5d624c2da735e72b387a12d9 [term 1 FOLLOWER]: Refusing update from remote peer 672a55e7d4874c5d99f3c5e3d9550f15: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:28.700937 27125 consensus_queue.cc:1035] T 450538b8217d40ce945d51e2c171ee36 P 672a55e7d4874c5d99f3c5e3d9550f15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:28.701341 27127 consensus_queue.cc:1035] T 450538b8217d40ce945d51e2c171ee36 P 672a55e7d4874c5d99f3c5e3d9550f15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:28.701848 27126 consensus_queue.cc:1035] T 450538b8217d40ce945d51e2c171ee36 P 672a55e7d4874c5d99f3c5e3d9550f15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:28.705525 27126 raft_consensus.cc:2932] T 450538b8217d40ce945d51e2c171ee36 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 75bd3f0e8d7c4e8aa92a353301afeb21 (127.9.176.194) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } attrs { replace: true } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { promote: false } } }
I20240502 01:55:28.713146 25000 raft_consensus.cc:2932] T 450538b8217d40ce945d51e2c171ee36 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 75bd3f0e8d7c4e8aa92a353301afeb21 (127.9.176.194) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } attrs { replace: true } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { promote: false } } }
I20240502 01:55:28.718327 26195 raft_consensus.cc:2932] T 450538b8217d40ce945d51e2c171ee36 P 6742b9db50c64ea394c2ca38ddee0872 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 75bd3f0e8d7c4e8aa92a353301afeb21 (127.9.176.194) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } attrs { replace: true } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { promote: false } } }
I20240502 01:55:28.719247 24725 catalog_manager.cc:5484] T 450538b8217d40ce945d51e2c171ee36 P 672a55e7d4874c5d99f3c5e3d9550f15 reported cstate change: config changed from index 7 to 8, 75bd3f0e8d7c4e8aa92a353301afeb21 (127.9.176.194) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:55:28.724787 24867 raft_consensus.cc:2932] T 450538b8217d40ce945d51e2c171ee36 P 96e5c2fd5d624c2da735e72b387a12d9 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 75bd3f0e8d7c4e8aa92a353301afeb21 (127.9.176.194) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } attrs { replace: true } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { promote: false } } }
I20240502 01:55:28.726929 25133 consensus_queue.cc:237] T 450538b8217d40ce945d51e2c171ee36 P 672a55e7d4874c5d99f3c5e3d9550f15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { promote: false } }
I20240502 01:55:28.729750 26195 raft_consensus.cc:1261] T 450538b8217d40ce945d51e2c171ee36 P 6742b9db50c64ea394c2ca38ddee0872 [term 1 FOLLOWER]: Refusing update from remote peer 672a55e7d4874c5d99f3c5e3d9550f15: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:28.729998 25000 raft_consensus.cc:1261] T 450538b8217d40ce945d51e2c171ee36 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 1 FOLLOWER]: Refusing update from remote peer 672a55e7d4874c5d99f3c5e3d9550f15: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:28.730383 27126 consensus_queue.cc:1035] T 450538b8217d40ce945d51e2c171ee36 P 672a55e7d4874c5d99f3c5e3d9550f15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:28.730736 27127 consensus_queue.cc:1035] T 450538b8217d40ce945d51e2c171ee36 P 672a55e7d4874c5d99f3c5e3d9550f15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:28.733458 27127 raft_consensus.cc:2932] T 450538b8217d40ce945d51e2c171ee36 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 96e5c2fd5d624c2da735e72b387a12d9 (127.9.176.193) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { promote: false } } }
I20240502 01:55:28.734176 26194 raft_consensus.cc:2932] T 450538b8217d40ce945d51e2c171ee36 P 6742b9db50c64ea394c2ca38ddee0872 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 96e5c2fd5d624c2da735e72b387a12d9 (127.9.176.193) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { promote: false } } }
I20240502 01:55:28.739347 25002 raft_consensus.cc:2932] T 450538b8217d40ce945d51e2c171ee36 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 96e5c2fd5d624c2da735e72b387a12d9 (127.9.176.193) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { promote: false } } }
I20240502 01:55:28.740792 24724 catalog_manager.cc:5484] T 450538b8217d40ce945d51e2c171ee36 P 6742b9db50c64ea394c2ca38ddee0872 reported cstate change: config changed from index 8 to 9, VOTER 96e5c2fd5d624c2da735e72b387a12d9 (127.9.176.193) evicted. New cstate: current_term: 1 leader_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { promote: false } } }
I20240502 01:55:28.743971 24712 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 450538b8217d40ce945d51e2c171ee36 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:55:28.745536 24848 tablet_service.cc:1514] Processing DeleteTablet for tablet 450538b8217d40ce945d51e2c171ee36 with delete_type TABLET_DATA_TOMBSTONED (TS 96e5c2fd5d624c2da735e72b387a12d9 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:54848
I20240502 01:55:28.746536 27240 tablet_replica.cc:330] T 450538b8217d40ce945d51e2c171ee36 P 96e5c2fd5d624c2da735e72b387a12d9: stopping tablet replica
I20240502 01:55:28.746834 27240 raft_consensus.cc:2229] T 450538b8217d40ce945d51e2c171ee36 P 96e5c2fd5d624c2da735e72b387a12d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:28.747352 27240 raft_consensus.cc:2258] T 450538b8217d40ce945d51e2c171ee36 P 96e5c2fd5d624c2da735e72b387a12d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:28.748746 27240 ts_tablet_manager.cc:1875] T 450538b8217d40ce945d51e2c171ee36 P 96e5c2fd5d624c2da735e72b387a12d9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:28.754770 27240 ts_tablet_manager.cc:1888] T 450538b8217d40ce945d51e2c171ee36 P 96e5c2fd5d624c2da735e72b387a12d9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 01:55:28.755066 27240 log.cc:1196] T 450538b8217d40ce945d51e2c171ee36 P 96e5c2fd5d624c2da735e72b387a12d9: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/wal/wals/450538b8217d40ce945d51e2c171ee36
I20240502 01:55:28.755947 24712 catalog_manager.cc:4830] TS 96e5c2fd5d624c2da735e72b387a12d9 (127.9.176.193:32783): tablet 450538b8217d40ce945d51e2c171ee36 (table rebalance_test_table_1 [id=4e02aab5efa847ab8735d911dc7a4f5a]) successfully deleted
I20240502 01:55:28.766904 26914 raft_consensus.cc:1050] T 32b0189cb19f42efbe7118003e9390f9 P 96e5c2fd5d624c2da735e72b387a12d9: attempting to promote NON_VOTER 672a55e7d4874c5d99f3c5e3d9550f15 to VOTER
I20240502 01:55:28.768172 26914 consensus_queue.cc:237] T 32b0189cb19f42efbe7118003e9390f9 P 96e5c2fd5d624c2da735e72b387a12d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { replace: true } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: false } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } attrs { promote: false } }
I20240502 01:55:28.770787 25933 raft_consensus.cc:1261] T 32b0189cb19f42efbe7118003e9390f9 P 30921110704f47f88d68a7d9364d42cc [term 1 FOLLOWER]: Refusing update from remote peer 96e5c2fd5d624c2da735e72b387a12d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:28.771073 25133 raft_consensus.cc:1261] T 32b0189cb19f42efbe7118003e9390f9 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 1 LEARNER]: Refusing update from remote peer 96e5c2fd5d624c2da735e72b387a12d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:28.771713 27216 consensus_queue.cc:1035] T 32b0189cb19f42efbe7118003e9390f9 P 96e5c2fd5d624c2da735e72b387a12d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:28.771970 25002 raft_consensus.cc:1261] T 32b0189cb19f42efbe7118003e9390f9 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 1 FOLLOWER]: Refusing update from remote peer 96e5c2fd5d624c2da735e72b387a12d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:28.772353 27216 consensus_queue.cc:1035] T 32b0189cb19f42efbe7118003e9390f9 P 96e5c2fd5d624c2da735e72b387a12d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.002s
I20240502 01:55:28.775120 27128 consensus_queue.cc:1035] T 32b0189cb19f42efbe7118003e9390f9 P 96e5c2fd5d624c2da735e72b387a12d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:28.780087 27216 raft_consensus.cc:2932] T 32b0189cb19f42efbe7118003e9390f9 P 96e5c2fd5d624c2da735e72b387a12d9 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 672a55e7d4874c5d99f3c5e3d9550f15 (127.9.176.195) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { replace: true } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: false } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } attrs { promote: false } } }
I20240502 01:55:28.784682 25133 raft_consensus.cc:2932] T 32b0189cb19f42efbe7118003e9390f9 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 672a55e7d4874c5d99f3c5e3d9550f15 (127.9.176.195) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { replace: true } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: false } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } attrs { promote: false } } }
I20240502 01:55:28.786243 24726 catalog_manager.cc:5484] T 32b0189cb19f42efbe7118003e9390f9 P 96e5c2fd5d624c2da735e72b387a12d9 reported cstate change: config changed from index 7 to 8, 672a55e7d4874c5d99f3c5e3d9550f15 (127.9.176.195) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "96e5c2fd5d624c2da735e72b387a12d9" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:55:28.788408 25933 raft_consensus.cc:2932] T 32b0189cb19f42efbe7118003e9390f9 P 30921110704f47f88d68a7d9364d42cc [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 672a55e7d4874c5d99f3c5e3d9550f15 (127.9.176.195) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { replace: true } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: false } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } attrs { promote: false } } }
I20240502 01:55:28.796026 24867 consensus_queue.cc:237] T 32b0189cb19f42efbe7118003e9390f9 P 96e5c2fd5d624c2da735e72b387a12d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: false } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } attrs { promote: false } }
I20240502 01:55:28.798823 25133 raft_consensus.cc:1261] T 32b0189cb19f42efbe7118003e9390f9 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 1 FOLLOWER]: Refusing update from remote peer 96e5c2fd5d624c2da735e72b387a12d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:28.799708 27129 consensus_queue.cc:1035] T 32b0189cb19f42efbe7118003e9390f9 P 96e5c2fd5d624c2da735e72b387a12d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:28.802947 27128 raft_consensus.cc:2932] T 32b0189cb19f42efbe7118003e9390f9 P 96e5c2fd5d624c2da735e72b387a12d9 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 75bd3f0e8d7c4e8aa92a353301afeb21 (127.9.176.194) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: false } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } attrs { promote: false } } }
I20240502 01:55:28.803449 25934 raft_consensus.cc:1261] T 32b0189cb19f42efbe7118003e9390f9 P 30921110704f47f88d68a7d9364d42cc [term 1 FOLLOWER]: Refusing update from remote peer 96e5c2fd5d624c2da735e72b387a12d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:28.803510 25134 raft_consensus.cc:2932] T 32b0189cb19f42efbe7118003e9390f9 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 75bd3f0e8d7c4e8aa92a353301afeb21 (127.9.176.194) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: false } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } attrs { promote: false } } }
I20240502 01:55:28.806710 26914 consensus_queue.cc:1035] T 32b0189cb19f42efbe7118003e9390f9 P 96e5c2fd5d624c2da735e72b387a12d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:28.808899 25933 raft_consensus.cc:2932] T 32b0189cb19f42efbe7118003e9390f9 P 30921110704f47f88d68a7d9364d42cc [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 75bd3f0e8d7c4e8aa92a353301afeb21 (127.9.176.194) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: false } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } attrs { promote: false } } }
I20240502 01:55:28.809762 24712 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 32b0189cb19f42efbe7118003e9390f9 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:55:28.810599 24726 catalog_manager.cc:5484] T 32b0189cb19f42efbe7118003e9390f9 P 672a55e7d4874c5d99f3c5e3d9550f15 reported cstate change: config changed from index 8 to 9, VOTER 75bd3f0e8d7c4e8aa92a353301afeb21 (127.9.176.194) evicted. New cstate: current_term: 1 leader_uuid: "96e5c2fd5d624c2da735e72b387a12d9" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: false } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } attrs { promote: false } } }
I20240502 01:55:28.825038 24982 tablet_service.cc:1514] Processing DeleteTablet for tablet 32b0189cb19f42efbe7118003e9390f9 with delete_type TABLET_DATA_TOMBSTONED (TS 75bd3f0e8d7c4e8aa92a353301afeb21 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:57512
I20240502 01:55:28.839792 27266 tablet_replica.cc:330] T 32b0189cb19f42efbe7118003e9390f9 P 75bd3f0e8d7c4e8aa92a353301afeb21: stopping tablet replica
I20240502 01:55:28.840309 27266 raft_consensus.cc:2229] T 32b0189cb19f42efbe7118003e9390f9 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:28.840677 27266 pending_rounds.cc:62] T 32b0189cb19f42efbe7118003e9390f9 P 75bd3f0e8d7c4e8aa92a353301afeb21: Trying to abort 1 pending ops.
I20240502 01:55:28.840842 27266 pending_rounds.cc:68] T 32b0189cb19f42efbe7118003e9390f9 P 75bd3f0e8d7c4e8aa92a353301afeb21: Aborting op as it isn't in flight: id { term: 1 index: 8 } timestamp: 7023062748231323648 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "32b0189cb19f42efbe7118003e9390f9" old_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { replace: true } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: false } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: NON_VOTER last_known_addr { host: "127.9.176.195" port: 46579 } attrs { promote: true } } } new_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { replace: true } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: false } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } attrs { promote: false } } } }
I20240502 01:55:28.841358 27266 raft_consensus.cc:2894] T 32b0189cb19f42efbe7118003e9390f9 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 1 FOLLOWER]: Aborting config change with OpId 1.8: Aborted: Op aborted
I20240502 01:55:28.841517 27266 raft_consensus.cc:2866] T 32b0189cb19f42efbe7118003e9390f9 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240502 01:55:28.841639 27266 raft_consensus.cc:2258] T 32b0189cb19f42efbe7118003e9390f9 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:28.843400 27266 ts_tablet_manager.cc:1875] T 32b0189cb19f42efbe7118003e9390f9 P 75bd3f0e8d7c4e8aa92a353301afeb21: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:28.850059 27266 ts_tablet_manager.cc:1888] T 32b0189cb19f42efbe7118003e9390f9 P 75bd3f0e8d7c4e8aa92a353301afeb21: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 01:55:28.850389 27266 log.cc:1196] T 32b0189cb19f42efbe7118003e9390f9 P 75bd3f0e8d7c4e8aa92a353301afeb21: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/wal/wals/32b0189cb19f42efbe7118003e9390f9
I20240502 01:55:28.852041 24712 catalog_manager.cc:4830] TS 75bd3f0e8d7c4e8aa92a353301afeb21 (127.9.176.194:41625): tablet 32b0189cb19f42efbe7118003e9390f9 (table rebalance_test_table_2 [id=26fb053e91c344cab36e6f0312de1a3c]) successfully deleted
I20240502 01:55:28.855610 27127 raft_consensus.cc:1050] T ce37ee2e946b4075b78f3c9456939848 P 672a55e7d4874c5d99f3c5e3d9550f15: attempting to promote NON_VOTER 96e5c2fd5d624c2da735e72b387a12d9 to VOTER
I20240502 01:55:28.856740 27127 consensus_queue.cc:237] T ce37ee2e946b4075b78f3c9456939848 P 672a55e7d4874c5d99f3c5e3d9550f15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { replace: true } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: false } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } attrs { promote: false } }
I20240502 01:55:28.860021 24867 raft_consensus.cc:1261] T ce37ee2e946b4075b78f3c9456939848 P 96e5c2fd5d624c2da735e72b387a12d9 [term 1 LEARNER]: Refusing update from remote peer 672a55e7d4874c5d99f3c5e3d9550f15: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:28.860211 26066 raft_consensus.cc:1261] T ce37ee2e946b4075b78f3c9456939848 P d490b54f0e954ef59deb6750523f6027 [term 1 FOLLOWER]: Refusing update from remote peer 672a55e7d4874c5d99f3c5e3d9550f15: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:28.860975 25002 raft_consensus.cc:1261] T ce37ee2e946b4075b78f3c9456939848 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 1 FOLLOWER]: Refusing update from remote peer 672a55e7d4874c5d99f3c5e3d9550f15: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:28.861063 27225 consensus_queue.cc:1035] T ce37ee2e946b4075b78f3c9456939848 P 672a55e7d4874c5d99f3c5e3d9550f15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:28.862030 27126 consensus_queue.cc:1035] T ce37ee2e946b4075b78f3c9456939848 P 672a55e7d4874c5d99f3c5e3d9550f15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:28.862699 27127 consensus_queue.cc:1035] T ce37ee2e946b4075b78f3c9456939848 P 672a55e7d4874c5d99f3c5e3d9550f15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:28.867537 27126 raft_consensus.cc:2932] T ce37ee2e946b4075b78f3c9456939848 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 96e5c2fd5d624c2da735e72b387a12d9 (127.9.176.193) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { replace: true } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: false } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } attrs { promote: false } } }
I20240502 01:55:28.868108 26066 raft_consensus.cc:2932] T ce37ee2e946b4075b78f3c9456939848 P d490b54f0e954ef59deb6750523f6027 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 96e5c2fd5d624c2da735e72b387a12d9 (127.9.176.193) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { replace: true } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: false } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } attrs { promote: false } } }
I20240502 01:55:28.868239 25002 raft_consensus.cc:2932] T ce37ee2e946b4075b78f3c9456939848 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 96e5c2fd5d624c2da735e72b387a12d9 (127.9.176.193) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { replace: true } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: false } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } attrs { promote: false } } }
I20240502 01:55:28.873180 24725 catalog_manager.cc:5484] T ce37ee2e946b4075b78f3c9456939848 P 75bd3f0e8d7c4e8aa92a353301afeb21 reported cstate change: config changed from index 7 to 8, 96e5c2fd5d624c2da735e72b387a12d9 (127.9.176.193) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { replace: true } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: false } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } attrs { promote: false } } }
I20240502 01:55:28.874369 24867 raft_consensus.cc:2932] T ce37ee2e946b4075b78f3c9456939848 P 96e5c2fd5d624c2da735e72b387a12d9 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 96e5c2fd5d624c2da735e72b387a12d9 (127.9.176.193) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { replace: true } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: false } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } attrs { promote: false } } }
I20240502 01:55:28.882489 25134 consensus_queue.cc:237] T ce37ee2e946b4075b78f3c9456939848 P 672a55e7d4874c5d99f3c5e3d9550f15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: false } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } attrs { promote: false } }
I20240502 01:55:28.885031 26066 raft_consensus.cc:1261] T ce37ee2e946b4075b78f3c9456939848 P d490b54f0e954ef59deb6750523f6027 [term 1 FOLLOWER]: Refusing update from remote peer 672a55e7d4874c5d99f3c5e3d9550f15: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:28.885031 24867 raft_consensus.cc:1261] T ce37ee2e946b4075b78f3c9456939848 P 96e5c2fd5d624c2da735e72b387a12d9 [term 1 FOLLOWER]: Refusing update from remote peer 672a55e7d4874c5d99f3c5e3d9550f15: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:28.885841 27225 consensus_queue.cc:1035] T ce37ee2e946b4075b78f3c9456939848 P 672a55e7d4874c5d99f3c5e3d9550f15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:28.886194 27126 consensus_queue.cc:1035] T ce37ee2e946b4075b78f3c9456939848 P 672a55e7d4874c5d99f3c5e3d9550f15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:28.888942 27127 raft_consensus.cc:2932] T ce37ee2e946b4075b78f3c9456939848 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 75bd3f0e8d7c4e8aa92a353301afeb21 (127.9.176.194) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: false } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } attrs { promote: false } } }
I20240502 01:55:28.890107 24867 raft_consensus.cc:2932] T ce37ee2e946b4075b78f3c9456939848 P 96e5c2fd5d624c2da735e72b387a12d9 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 75bd3f0e8d7c4e8aa92a353301afeb21 (127.9.176.194) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: false } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } attrs { promote: false } } }
I20240502 01:55:28.890259 26066 raft_consensus.cc:2932] T ce37ee2e946b4075b78f3c9456939848 P d490b54f0e954ef59deb6750523f6027 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 75bd3f0e8d7c4e8aa92a353301afeb21 (127.9.176.194) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: false } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } attrs { promote: false } } }
I20240502 01:55:28.894304 24712 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ce37ee2e946b4075b78f3c9456939848 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:55:28.895625 24722 catalog_manager.cc:5484] T ce37ee2e946b4075b78f3c9456939848 P 672a55e7d4874c5d99f3c5e3d9550f15 reported cstate change: config changed from index 8 to 9, VOTER 75bd3f0e8d7c4e8aa92a353301afeb21 (127.9.176.194) evicted. New cstate: current_term: 1 leader_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:55:28.901997 24982 tablet_service.cc:1514] Processing DeleteTablet for tablet ce37ee2e946b4075b78f3c9456939848 with delete_type TABLET_DATA_TOMBSTONED (TS 75bd3f0e8d7c4e8aa92a353301afeb21 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:57512
I20240502 01:55:28.902536 27266 tablet_replica.cc:330] T ce37ee2e946b4075b78f3c9456939848 P 75bd3f0e8d7c4e8aa92a353301afeb21: stopping tablet replica
I20240502 01:55:28.902752 27266 raft_consensus.cc:2229] T ce37ee2e946b4075b78f3c9456939848 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:28.903187 27266 raft_consensus.cc:2258] T ce37ee2e946b4075b78f3c9456939848 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:28.904588 27266 ts_tablet_manager.cc:1875] T ce37ee2e946b4075b78f3c9456939848 P 75bd3f0e8d7c4e8aa92a353301afeb21: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:28.910125 27266 ts_tablet_manager.cc:1888] T ce37ee2e946b4075b78f3c9456939848 P 75bd3f0e8d7c4e8aa92a353301afeb21: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 01:55:28.910277 27266 log.cc:1196] T ce37ee2e946b4075b78f3c9456939848 P 75bd3f0e8d7c4e8aa92a353301afeb21: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/wal/wals/ce37ee2e946b4075b78f3c9456939848
I20240502 01:55:28.911085 24712 catalog_manager.cc:4830] TS 75bd3f0e8d7c4e8aa92a353301afeb21 (127.9.176.194:41625): tablet ce37ee2e946b4075b78f3c9456939848 (table rebalance_test_table_0 [id=a39bf554fb4649d8983f51465c4cd34d]) successfully deleted
I20240502 01:55:28.951118 27199 raft_consensus.cc:1050] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 75bd3f0e8d7c4e8aa92a353301afeb21: attempting to promote NON_VOTER 6742b9db50c64ea394c2ca38ddee0872 to VOTER
I20240502 01:55:28.952066 27199 consensus_queue.cc:237] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 75bd3f0e8d7c4e8aa92a353301afeb21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: false replace: true } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false } }
I20240502 01:55:28.954824 26194 raft_consensus.cc:1261] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 6742b9db50c64ea394c2ca38ddee0872 [term 1 LEARNER]: Refusing update from remote peer 75bd3f0e8d7c4e8aa92a353301afeb21: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:28.955988 26066 raft_consensus.cc:1261] T 1b76f351e8644d9e8ff27e3f1a963ba2 P d490b54f0e954ef59deb6750523f6027 [term 1 FOLLOWER]: Refusing update from remote peer 75bd3f0e8d7c4e8aa92a353301afeb21: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:28.956687 27199 consensus_queue.cc:1035] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 75bd3f0e8d7c4e8aa92a353301afeb21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:28.956048 25134 raft_consensus.cc:1261] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 1 FOLLOWER]: Refusing update from remote peer 75bd3f0e8d7c4e8aa92a353301afeb21: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:28.957823 27198 consensus_queue.cc:1035] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 75bd3f0e8d7c4e8aa92a353301afeb21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:28.961246 27199 consensus_queue.cc:1035] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 75bd3f0e8d7c4e8aa92a353301afeb21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:28.964711 27198 raft_consensus.cc:2932] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6742b9db50c64ea394c2ca38ddee0872 (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: false replace: true } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false } } }
I20240502 01:55:28.966138 26194 raft_consensus.cc:2932] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 6742b9db50c64ea394c2ca38ddee0872 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6742b9db50c64ea394c2ca38ddee0872 (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: false replace: true } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false } } }
I20240502 01:55:28.970100 26066 raft_consensus.cc:2932] T 1b76f351e8644d9e8ff27e3f1a963ba2 P d490b54f0e954ef59deb6750523f6027 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6742b9db50c64ea394c2ca38ddee0872 (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: false replace: true } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false } } }
I20240502 01:55:28.974174 24722 catalog_manager.cc:5484] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 6742b9db50c64ea394c2ca38ddee0872 reported cstate change: config changed from index 7 to 8, 6742b9db50c64ea394c2ca38ddee0872 (127.9.176.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: false replace: true } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false } } }
I20240502 01:55:28.976244 25134 raft_consensus.cc:2932] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6742b9db50c64ea394c2ca38ddee0872 (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: false replace: true } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false } } }
I20240502 01:55:28.984839 25002 consensus_queue.cc:237] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 75bd3f0e8d7c4e8aa92a353301afeb21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false } }
I20240502 01:55:28.987249 25134 raft_consensus.cc:1261] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 1 FOLLOWER]: Refusing update from remote peer 75bd3f0e8d7c4e8aa92a353301afeb21: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:28.988284 27199 consensus_queue.cc:1035] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 75bd3f0e8d7c4e8aa92a353301afeb21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:28.990199 26194 raft_consensus.cc:1261] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 6742b9db50c64ea394c2ca38ddee0872 [term 1 FOLLOWER]: Refusing update from remote peer 75bd3f0e8d7c4e8aa92a353301afeb21: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:28.990929 27199 consensus_queue.cc:1035] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 75bd3f0e8d7c4e8aa92a353301afeb21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:28.993995 27198 raft_consensus.cc:2932] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER d490b54f0e954ef59deb6750523f6027 (127.9.176.198) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false } } }
I20240502 01:55:28.994731 25134 raft_consensus.cc:2932] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER d490b54f0e954ef59deb6750523f6027 (127.9.176.198) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false } } }
I20240502 01:55:28.998225 24712 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1b76f351e8644d9e8ff27e3f1a963ba2 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:55:29.001096 24724 catalog_manager.cc:5484] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 672a55e7d4874c5d99f3c5e3d9550f15 reported cstate change: config changed from index 8 to 9, VOTER d490b54f0e954ef59deb6750523f6027 (127.9.176.198) evicted. New cstate: current_term: 1 leader_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false } } }
I20240502 01:55:29.002941 26194 raft_consensus.cc:2932] T 1b76f351e8644d9e8ff27e3f1a963ba2 P 6742b9db50c64ea394c2ca38ddee0872 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER d490b54f0e954ef59deb6750523f6027 (127.9.176.198) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false } } }
I20240502 01:55:29.027031 25914 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:29.035935 25112 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:55:29.037292 26045 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b76f351e8644d9e8ff27e3f1a963ba2 with delete_type TABLET_DATA_TOMBSTONED (TS d490b54f0e954ef59deb6750523f6027 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:53668
I20240502 01:55:29.040678 26177 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:29.055209 27324 tablet_replica.cc:330] T 1b76f351e8644d9e8ff27e3f1a963ba2 P d490b54f0e954ef59deb6750523f6027: stopping tablet replica
I20240502 01:55:29.058089 27127 raft_consensus.cc:1050] T b6beb3b47fd545d2b6a2d9181f4e5468 P 672a55e7d4874c5d99f3c5e3d9550f15: attempting to promote NON_VOTER d490b54f0e954ef59deb6750523f6027 to VOTER
I20240502 01:55:29.059223 27127 consensus_queue.cc:237] T b6beb3b47fd545d2b6a2d9181f4e5468 P 672a55e7d4874c5d99f3c5e3d9550f15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false replace: true } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: false } }
I20240502 01:55:29.065114 27324 raft_consensus.cc:2229] T 1b76f351e8644d9e8ff27e3f1a963ba2 P d490b54f0e954ef59deb6750523f6027 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:29.073704 24982 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:55:29.087177 26045 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:29.095757 27198 raft_consensus.cc:1050] T 08bb5eb2c2814019b6eda6fbbc288245 P 75bd3f0e8d7c4e8aa92a353301afeb21: attempting to promote NON_VOTER 6742b9db50c64ea394c2ca38ddee0872 to VOTER
I20240502 01:55:29.097206 27198 consensus_queue.cc:237] T 08bb5eb2c2814019b6eda6fbbc288245 P 75bd3f0e8d7c4e8aa92a353301afeb21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: false replace: true } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false } }
I20240502 01:55:29.103753 27324 raft_consensus.cc:2258] T 1b76f351e8644d9e8ff27e3f1a963ba2 P d490b54f0e954ef59deb6750523f6027 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:29.106280 27324 ts_tablet_manager.cc:1875] T 1b76f351e8644d9e8ff27e3f1a963ba2 P d490b54f0e954ef59deb6750523f6027: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:29.107792 26194 raft_consensus.cc:1261] T 08bb5eb2c2814019b6eda6fbbc288245 P 6742b9db50c64ea394c2ca38ddee0872 [term 1 LEARNER]: Refusing update from remote peer 75bd3f0e8d7c4e8aa92a353301afeb21: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:29.108701 27000 consensus_queue.cc:1035] T 08bb5eb2c2814019b6eda6fbbc288245 P 75bd3f0e8d7c4e8aa92a353301afeb21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:29.113651 25933 raft_consensus.cc:1261] T 08bb5eb2c2814019b6eda6fbbc288245 P 30921110704f47f88d68a7d9364d42cc [term 1 FOLLOWER]: Refusing update from remote peer 75bd3f0e8d7c4e8aa92a353301afeb21: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:29.114432 27000 consensus_queue.cc:1035] T 08bb5eb2c2814019b6eda6fbbc288245 P 75bd3f0e8d7c4e8aa92a353301afeb21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:29.114650 27324 ts_tablet_manager.cc:1888] T 1b76f351e8644d9e8ff27e3f1a963ba2 P d490b54f0e954ef59deb6750523f6027: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 01:55:29.114825 27324 log.cc:1196] T 1b76f351e8644d9e8ff27e3f1a963ba2 P d490b54f0e954ef59deb6750523f6027: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-5/wal/wals/1b76f351e8644d9e8ff27e3f1a963ba2
I20240502 01:55:29.123139 24848 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:55:29.127511 24713 catalog_manager.cc:4830] TS d490b54f0e954ef59deb6750523f6027 (127.9.176.198:34317): tablet 1b76f351e8644d9e8ff27e3f1a963ba2 (table rebalance_test_table_0 [id=a39bf554fb4649d8983f51465c4cd34d]) successfully deleted
I20240502 01:55:29.128592 24867 raft_consensus.cc:1261] T 08bb5eb2c2814019b6eda6fbbc288245 P 96e5c2fd5d624c2da735e72b387a12d9 [term 1 FOLLOWER]: Refusing update from remote peer 75bd3f0e8d7c4e8aa92a353301afeb21: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:29.129261 27000 consensus_queue.cc:1035] T 08bb5eb2c2814019b6eda6fbbc288245 P 75bd3f0e8d7c4e8aa92a353301afeb21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:29.134377 26195 raft_consensus.cc:1261] T b6beb3b47fd545d2b6a2d9181f4e5468 P 6742b9db50c64ea394c2ca38ddee0872 [term 1 FOLLOWER]: Refusing update from remote peer 672a55e7d4874c5d99f3c5e3d9550f15: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:29.135167 27126 consensus_queue.cc:1035] T b6beb3b47fd545d2b6a2d9181f4e5468 P 672a55e7d4874c5d99f3c5e3d9550f15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:29.142939 27199 raft_consensus.cc:2932] T 08bb5eb2c2814019b6eda6fbbc288245 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6742b9db50c64ea394c2ca38ddee0872 (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: false replace: true } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false } } }
I20240502 01:55:29.147621 26195 raft_consensus.cc:2932] T 08bb5eb2c2814019b6eda6fbbc288245 P 6742b9db50c64ea394c2ca38ddee0872 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6742b9db50c64ea394c2ca38ddee0872 (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: false replace: true } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false } } }
I20240502 01:55:29.150869 24722 catalog_manager.cc:5484] T 08bb5eb2c2814019b6eda6fbbc288245 P 75bd3f0e8d7c4e8aa92a353301afeb21 reported cstate change: config changed from index 7 to 8, 6742b9db50c64ea394c2ca38ddee0872 (127.9.176.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: false replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:55:29.153921 25933 raft_consensus.cc:2932] T 08bb5eb2c2814019b6eda6fbbc288245 P 30921110704f47f88d68a7d9364d42cc [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6742b9db50c64ea394c2ca38ddee0872 (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: false replace: true } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false } } }
I20240502 01:55:29.161568 26066 raft_consensus.cc:1261] T b6beb3b47fd545d2b6a2d9181f4e5468 P d490b54f0e954ef59deb6750523f6027 [term 1 LEARNER]: Refusing update from remote peer 672a55e7d4874c5d99f3c5e3d9550f15: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:29.162284 27126 consensus_queue.cc:1035] T b6beb3b47fd545d2b6a2d9181f4e5468 P 672a55e7d4874c5d99f3c5e3d9550f15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:29.163558 24869 raft_consensus.cc:1261] T b6beb3b47fd545d2b6a2d9181f4e5468 P 96e5c2fd5d624c2da735e72b387a12d9 [term 1 FOLLOWER]: Refusing update from remote peer 672a55e7d4874c5d99f3c5e3d9550f15: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:29.164573 27126 consensus_queue.cc:1035] T b6beb3b47fd545d2b6a2d9181f4e5468 P 672a55e7d4874c5d99f3c5e3d9550f15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:29.166091 27126 raft_consensus.cc:2932] T b6beb3b47fd545d2b6a2d9181f4e5468 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d490b54f0e954ef59deb6750523f6027 (127.9.176.198) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false replace: true } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: false } } }
I20240502 01:55:29.172564 26066 raft_consensus.cc:2932] T b6beb3b47fd545d2b6a2d9181f4e5468 P d490b54f0e954ef59deb6750523f6027 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d490b54f0e954ef59deb6750523f6027 (127.9.176.198) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false replace: true } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: false } } }
I20240502 01:55:29.180665 24724 catalog_manager.cc:5484] T b6beb3b47fd545d2b6a2d9181f4e5468 P d490b54f0e954ef59deb6750523f6027 reported cstate change: config changed from index 7 to 8, d490b54f0e954ef59deb6750523f6027 (127.9.176.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false replace: true } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: false } } }
I20240502 01:55:29.186502 26194 raft_consensus.cc:2932] T b6beb3b47fd545d2b6a2d9181f4e5468 P 6742b9db50c64ea394c2ca38ddee0872 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d490b54f0e954ef59deb6750523f6027 (127.9.176.198) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false replace: true } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: false } } }
I20240502 01:55:29.229749 25002 consensus_queue.cc:237] T 08bb5eb2c2814019b6eda6fbbc288245 P 75bd3f0e8d7c4e8aa92a353301afeb21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false } }
I20240502 01:55:29.232607 24867 raft_consensus.cc:1261] T 08bb5eb2c2814019b6eda6fbbc288245 P 96e5c2fd5d624c2da735e72b387a12d9 [term 1 FOLLOWER]: Refusing update from remote peer 75bd3f0e8d7c4e8aa92a353301afeb21: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:29.234405 27000 consensus_queue.cc:1035] T 08bb5eb2c2814019b6eda6fbbc288245 P 75bd3f0e8d7c4e8aa92a353301afeb21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20240502 01:55:29.237031 24868 raft_consensus.cc:2932] T b6beb3b47fd545d2b6a2d9181f4e5468 P 96e5c2fd5d624c2da735e72b387a12d9 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d490b54f0e954ef59deb6750523f6027 (127.9.176.198) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false replace: true } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: false } } }
I20240502 01:55:29.240164 24867 raft_consensus.cc:2932] T 08bb5eb2c2814019b6eda6fbbc288245 P 96e5c2fd5d624c2da735e72b387a12d9 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6742b9db50c64ea394c2ca38ddee0872 (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: false replace: true } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false } } }
I20240502 01:55:29.241809 25134 consensus_queue.cc:237] T b6beb3b47fd545d2b6a2d9181f4e5468 P 672a55e7d4874c5d99f3c5e3d9550f15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: false } }
I20240502 01:55:29.243808 26194 raft_consensus.cc:1261] T 08bb5eb2c2814019b6eda6fbbc288245 P 6742b9db50c64ea394c2ca38ddee0872 [term 1 FOLLOWER]: Refusing update from remote peer 75bd3f0e8d7c4e8aa92a353301afeb21: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:29.245054 27000 consensus_queue.cc:1035] T 08bb5eb2c2814019b6eda6fbbc288245 P 75bd3f0e8d7c4e8aa92a353301afeb21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:29.248633 24868 raft_consensus.cc:1261] T b6beb3b47fd545d2b6a2d9181f4e5468 P 96e5c2fd5d624c2da735e72b387a12d9 [term 1 FOLLOWER]: Refusing update from remote peer 672a55e7d4874c5d99f3c5e3d9550f15: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:29.249321 26066 raft_consensus.cc:1261] T b6beb3b47fd545d2b6a2d9181f4e5468 P d490b54f0e954ef59deb6750523f6027 [term 1 FOLLOWER]: Refusing update from remote peer 672a55e7d4874c5d99f3c5e3d9550f15: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:29.249411 27198 raft_consensus.cc:2932] T 08bb5eb2c2814019b6eda6fbbc288245 P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 30921110704f47f88d68a7d9364d42cc (127.9.176.196) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false } } }
I20240502 01:55:29.250267 27125 consensus_queue.cc:1035] T b6beb3b47fd545d2b6a2d9181f4e5468 P 672a55e7d4874c5d99f3c5e3d9550f15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:29.250849 27125 consensus_queue.cc:1035] T b6beb3b47fd545d2b6a2d9181f4e5468 P 672a55e7d4874c5d99f3c5e3d9550f15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:29.253482 27125 raft_consensus.cc:2932] T b6beb3b47fd545d2b6a2d9181f4e5468 P 672a55e7d4874c5d99f3c5e3d9550f15 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 6742b9db50c64ea394c2ca38ddee0872 (127.9.176.197) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: false } } }
I20240502 01:55:29.254554 26194 raft_consensus.cc:2932] T 08bb5eb2c2814019b6eda6fbbc288245 P 6742b9db50c64ea394c2ca38ddee0872 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 30921110704f47f88d68a7d9364d42cc (127.9.176.196) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false } } }
I20240502 01:55:29.259099 24712 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 08bb5eb2c2814019b6eda6fbbc288245 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:55:29.261492 24724 catalog_manager.cc:5484] T 08bb5eb2c2814019b6eda6fbbc288245 P 6742b9db50c64ea394c2ca38ddee0872 reported cstate change: config changed from index 8 to 9, VOTER 30921110704f47f88d68a7d9364d42cc (127.9.176.196) evicted. New cstate: current_term: 1 leader_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false } } }
I20240502 01:55:29.262149 24869 raft_consensus.cc:2932] T 08bb5eb2c2814019b6eda6fbbc288245 P 96e5c2fd5d624c2da735e72b387a12d9 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 30921110704f47f88d68a7d9364d42cc (127.9.176.196) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } } peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false } } }
I20240502 01:55:29.263038 24867 raft_consensus.cc:2932] T b6beb3b47fd545d2b6a2d9181f4e5468 P 96e5c2fd5d624c2da735e72b387a12d9 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 6742b9db50c64ea394c2ca38ddee0872 (127.9.176.197) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: false } } }
I20240502 01:55:29.265486 24726 catalog_manager.cc:5484] T b6beb3b47fd545d2b6a2d9181f4e5468 P 672a55e7d4874c5d99f3c5e3d9550f15 reported cstate change: config changed from index 8 to 9, VOTER 6742b9db50c64ea394c2ca38ddee0872 (127.9.176.197) evicted. New cstate: current_term: 1 leader_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:55:29.268447 24712 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b6beb3b47fd545d2b6a2d9181f4e5468 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:55:29.270562 26066 raft_consensus.cc:2932] T b6beb3b47fd545d2b6a2d9181f4e5468 P d490b54f0e954ef59deb6750523f6027 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 6742b9db50c64ea394c2ca38ddee0872 (127.9.176.197) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "672a55e7d4874c5d99f3c5e3d9550f15" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46579 } } peers { permanent_uuid: "d490b54f0e954ef59deb6750523f6027" member_type: VOTER last_known_addr { host: "127.9.176.198" port: 34317 } attrs { promote: false } } }
I20240502 01:55:29.280663 25914 tablet_service.cc:1514] Processing DeleteTablet for tablet 08bb5eb2c2814019b6eda6fbbc288245 with delete_type TABLET_DATA_TOMBSTONED (TS 30921110704f47f88d68a7d9364d42cc not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:60780
I20240502 01:55:29.290886 27342 tablet_replica.cc:330] T 08bb5eb2c2814019b6eda6fbbc288245 P 30921110704f47f88d68a7d9364d42cc: stopping tablet replica
I20240502 01:55:29.291548 27342 raft_consensus.cc:2229] T 08bb5eb2c2814019b6eda6fbbc288245 P 30921110704f47f88d68a7d9364d42cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:29.292088 27342 raft_consensus.cc:2258] T 08bb5eb2c2814019b6eda6fbbc288245 P 30921110704f47f88d68a7d9364d42cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:29.292408 26177 tablet_service.cc:1514] Processing DeleteTablet for tablet b6beb3b47fd545d2b6a2d9181f4e5468 with delete_type TABLET_DATA_TOMBSTONED (TS 6742b9db50c64ea394c2ca38ddee0872 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:43780
I20240502 01:55:29.293710 27342 ts_tablet_manager.cc:1875] T 08bb5eb2c2814019b6eda6fbbc288245 P 30921110704f47f88d68a7d9364d42cc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:29.295607 27348 tablet_replica.cc:330] T b6beb3b47fd545d2b6a2d9181f4e5468 P 6742b9db50c64ea394c2ca38ddee0872: stopping tablet replica
I20240502 01:55:29.296198 27348 raft_consensus.cc:2229] T b6beb3b47fd545d2b6a2d9181f4e5468 P 6742b9db50c64ea394c2ca38ddee0872 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:29.297056 27348 raft_consensus.cc:2258] T b6beb3b47fd545d2b6a2d9181f4e5468 P 6742b9db50c64ea394c2ca38ddee0872 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:29.298677 27348 ts_tablet_manager.cc:1875] T b6beb3b47fd545d2b6a2d9181f4e5468 P 6742b9db50c64ea394c2ca38ddee0872: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:29.301632 27342 ts_tablet_manager.cc:1888] T 08bb5eb2c2814019b6eda6fbbc288245 P 30921110704f47f88d68a7d9364d42cc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 01:55:29.301868 27342 log.cc:1196] T 08bb5eb2c2814019b6eda6fbbc288245 P 30921110704f47f88d68a7d9364d42cc: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/wal/wals/08bb5eb2c2814019b6eda6fbbc288245
I20240502 01:55:29.302958 24714 catalog_manager.cc:4830] TS 30921110704f47f88d68a7d9364d42cc (127.9.176.196:39355): tablet 08bb5eb2c2814019b6eda6fbbc288245 (table rebalance_test_table_2 [id=26fb053e91c344cab36e6f0312de1a3c]) successfully deleted
I20240502 01:55:29.306099 27348 ts_tablet_manager.cc:1888] T b6beb3b47fd545d2b6a2d9181f4e5468 P 6742b9db50c64ea394c2ca38ddee0872: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 01:55:29.306363 27348 log.cc:1196] T b6beb3b47fd545d2b6a2d9181f4e5468 P 6742b9db50c64ea394c2ca38ddee0872: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/wal/wals/b6beb3b47fd545d2b6a2d9181f4e5468
I20240502 01:55:29.308117 24711 catalog_manager.cc:4830] TS 6742b9db50c64ea394c2ca38ddee0872 (127.9.176.197:43479): tablet b6beb3b47fd545d2b6a2d9181f4e5468 (table rebalance_test_table_1 [id=4e02aab5efa847ab8735d911dc7a4f5a]) successfully deleted
I20240502 01:55:29.406581 25914 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:29.425514 25112 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:55:29.436379 26045 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:29.438726 26177 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:29.469413 24982 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:55:29.474390 24848 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:55:29.553431 24867 consensus_queue.cc:237] T 75d71cdcf4d44353acb7d2f18696977b P 96e5c2fd5d624c2da735e72b387a12d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false replace: true } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { promote: false } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: true } }
I20240502 01:55:29.555524 25002 raft_consensus.cc:1261] T 75d71cdcf4d44353acb7d2f18696977b P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 1 FOLLOWER]: Refusing update from remote peer 96e5c2fd5d624c2da735e72b387a12d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:29.555636 26194 raft_consensus.cc:1261] T 75d71cdcf4d44353acb7d2f18696977b P 6742b9db50c64ea394c2ca38ddee0872 [term 1 FOLLOWER]: Refusing update from remote peer 96e5c2fd5d624c2da735e72b387a12d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
W20240502 01:55:29.555972 24805 consensus_peers.cc:485] T 75d71cdcf4d44353acb7d2f18696977b P 96e5c2fd5d624c2da735e72b387a12d9 -> Peer 30921110704f47f88d68a7d9364d42cc (127.9.176.196:39355): Couldn't send request to peer 30921110704f47f88d68a7d9364d42cc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 75d71cdcf4d44353acb7d2f18696977b. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:29.556191 27128 consensus_queue.cc:1035] T 75d71cdcf4d44353acb7d2f18696977b P 96e5c2fd5d624c2da735e72b387a12d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:29.556485 26914 consensus_queue.cc:1035] T 75d71cdcf4d44353acb7d2f18696977b P 96e5c2fd5d624c2da735e72b387a12d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:29.558719 27129 raft_consensus.cc:2932] T 75d71cdcf4d44353acb7d2f18696977b P 96e5c2fd5d624c2da735e72b387a12d9 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 30921110704f47f88d68a7d9364d42cc (127.9.176.196) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false replace: true } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { promote: false } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: true } } }
I20240502 01:55:29.559314 25002 raft_consensus.cc:2932] T 75d71cdcf4d44353acb7d2f18696977b P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 30921110704f47f88d68a7d9364d42cc (127.9.176.196) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false replace: true } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { promote: false } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: true } } }
I20240502 01:55:29.559429 26194 raft_consensus.cc:2932] T 75d71cdcf4d44353acb7d2f18696977b P 6742b9db50c64ea394c2ca38ddee0872 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 30921110704f47f88d68a7d9364d42cc (127.9.176.196) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false replace: true } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { promote: false } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: true } } }
I20240502 01:55:29.563983 24726 catalog_manager.cc:5484] T 75d71cdcf4d44353acb7d2f18696977b P 6742b9db50c64ea394c2ca38ddee0872 reported cstate change: config changed from index 9 to 10, NON_VOTER 30921110704f47f88d68a7d9364d42cc (127.9.176.196) added. New cstate: current_term: 1 leader_uuid: "96e5c2fd5d624c2da735e72b387a12d9" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false replace: true } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { promote: false } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: true } } }
I20240502 01:55:30.027381 27403 ts_tablet_manager.cc:902] T 75d71cdcf4d44353acb7d2f18696977b P 30921110704f47f88d68a7d9364d42cc: Initiating tablet copy from peer 96e5c2fd5d624c2da735e72b387a12d9 (127.9.176.193:32783)
I20240502 01:55:30.028635 27403 tablet_copy_client.cc:308] T 75d71cdcf4d44353acb7d2f18696977b P 30921110704f47f88d68a7d9364d42cc: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.193:32783
I20240502 01:55:30.029438 24889 tablet_copy_service.cc:140] P 96e5c2fd5d624c2da735e72b387a12d9: Received BeginTabletCopySession request for tablet 75d71cdcf4d44353acb7d2f18696977b from peer 30921110704f47f88d68a7d9364d42cc ({username='slave'} at 127.9.176.196:36527)
I20240502 01:55:30.029609 24889 tablet_copy_service.cc:161] P 96e5c2fd5d624c2da735e72b387a12d9: Beginning new tablet copy session on tablet 75d71cdcf4d44353acb7d2f18696977b from peer 30921110704f47f88d68a7d9364d42cc at {username='slave'} at 127.9.176.196:36527: session id = 30921110704f47f88d68a7d9364d42cc-75d71cdcf4d44353acb7d2f18696977b
I20240502 01:55:30.031558 24889 tablet_copy_source_session.cc:205] T 75d71cdcf4d44353acb7d2f18696977b P 96e5c2fd5d624c2da735e72b387a12d9: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:30.032864 27403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75d71cdcf4d44353acb7d2f18696977b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:30.037674 27403 tablet_copy_client.cc:783] T 75d71cdcf4d44353acb7d2f18696977b P 30921110704f47f88d68a7d9364d42cc: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:30.037950 27403 tablet_copy_client.cc:647] T 75d71cdcf4d44353acb7d2f18696977b P 30921110704f47f88d68a7d9364d42cc: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:30.039883 27403 tablet_copy_client.cc:523] T 75d71cdcf4d44353acb7d2f18696977b P 30921110704f47f88d68a7d9364d42cc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:30.042344 27403 tablet_bootstrap.cc:492] T 75d71cdcf4d44353acb7d2f18696977b P 30921110704f47f88d68a7d9364d42cc: Bootstrap starting.
I20240502 01:55:30.105535 27403 tablet_bootstrap.cc:492] T 75d71cdcf4d44353acb7d2f18696977b P 30921110704f47f88d68a7d9364d42cc: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:30.106159 27403 tablet_bootstrap.cc:492] T 75d71cdcf4d44353acb7d2f18696977b P 30921110704f47f88d68a7d9364d42cc: Bootstrap complete.
I20240502 01:55:30.106557 27403 ts_tablet_manager.cc:1367] T 75d71cdcf4d44353acb7d2f18696977b P 30921110704f47f88d68a7d9364d42cc: Time spent bootstrapping tablet: real 0.064s	user 0.055s	sys 0.008s
I20240502 01:55:30.107729 27403 raft_consensus.cc:348] T 75d71cdcf4d44353acb7d2f18696977b P 30921110704f47f88d68a7d9364d42cc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false replace: true } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { promote: false } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: true } }
I20240502 01:55:30.108106 27403 raft_consensus.cc:729] T 75d71cdcf4d44353acb7d2f18696977b P 30921110704f47f88d68a7d9364d42cc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 30921110704f47f88d68a7d9364d42cc, State: Initialized, Role: LEARNER
I20240502 01:55:30.108376 27403 consensus_queue.cc:260] T 75d71cdcf4d44353acb7d2f18696977b P 30921110704f47f88d68a7d9364d42cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false replace: true } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { promote: false } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: true } }
I20240502 01:55:30.109519 27403 ts_tablet_manager.cc:1398] T 75d71cdcf4d44353acb7d2f18696977b P 30921110704f47f88d68a7d9364d42cc: Time spent starting tablet: real 0.003s	user 0.005s	sys 0.000s
I20240502 01:55:30.110800 24889 tablet_copy_service.cc:342] P 96e5c2fd5d624c2da735e72b387a12d9: Request end of tablet copy session 30921110704f47f88d68a7d9364d42cc-75d71cdcf4d44353acb7d2f18696977b received from {username='slave'} at 127.9.176.196:36527
I20240502 01:55:30.110955 24889 tablet_copy_service.cc:433] P 96e5c2fd5d624c2da735e72b387a12d9: ending tablet copy session 30921110704f47f88d68a7d9364d42cc-75d71cdcf4d44353acb7d2f18696977b on tablet 75d71cdcf4d44353acb7d2f18696977b with peer 30921110704f47f88d68a7d9364d42cc
I20240502 01:55:30.541378 25933 raft_consensus.cc:1203] T 75d71cdcf4d44353acb7d2f18696977b P 30921110704f47f88d68a7d9364d42cc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10]   Dedup: 1.10->[]
I20240502 01:55:31.139490 27437 raft_consensus.cc:1050] T 75d71cdcf4d44353acb7d2f18696977b P 96e5c2fd5d624c2da735e72b387a12d9: attempting to promote NON_VOTER 30921110704f47f88d68a7d9364d42cc to VOTER
I20240502 01:55:31.140281 27437 consensus_queue.cc:237] T 75d71cdcf4d44353acb7d2f18696977b P 96e5c2fd5d624c2da735e72b387a12d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false replace: true } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { promote: false } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: false } }
I20240502 01:55:31.142804 25933 raft_consensus.cc:1261] T 75d71cdcf4d44353acb7d2f18696977b P 30921110704f47f88d68a7d9364d42cc [term 1 LEARNER]: Refusing update from remote peer 96e5c2fd5d624c2da735e72b387a12d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 01:55:31.142993 25002 raft_consensus.cc:1261] T 75d71cdcf4d44353acb7d2f18696977b P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 1 FOLLOWER]: Refusing update from remote peer 96e5c2fd5d624c2da735e72b387a12d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 01:55:31.143391 26194 raft_consensus.cc:1261] T 75d71cdcf4d44353acb7d2f18696977b P 6742b9db50c64ea394c2ca38ddee0872 [term 1 FOLLOWER]: Refusing update from remote peer 96e5c2fd5d624c2da735e72b387a12d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 01:55:31.143415 27421 consensus_queue.cc:1035] T 75d71cdcf4d44353acb7d2f18696977b P 96e5c2fd5d624c2da735e72b387a12d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 01:55:31.143759 27437 consensus_queue.cc:1035] T 75d71cdcf4d44353acb7d2f18696977b P 96e5c2fd5d624c2da735e72b387a12d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 01:55:31.144047 27440 consensus_queue.cc:1035] T 75d71cdcf4d44353acb7d2f18696977b P 96e5c2fd5d624c2da735e72b387a12d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 01:55:31.147507 27440 raft_consensus.cc:2932] T 75d71cdcf4d44353acb7d2f18696977b P 96e5c2fd5d624c2da735e72b387a12d9 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 30921110704f47f88d68a7d9364d42cc (127.9.176.196) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false replace: true } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { promote: false } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: false } } }
I20240502 01:55:31.148258 25002 raft_consensus.cc:2932] T 75d71cdcf4d44353acb7d2f18696977b P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 30921110704f47f88d68a7d9364d42cc (127.9.176.196) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false replace: true } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { promote: false } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: false } } }
I20240502 01:55:31.148260 25933 raft_consensus.cc:2932] T 75d71cdcf4d44353acb7d2f18696977b P 30921110704f47f88d68a7d9364d42cc [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 30921110704f47f88d68a7d9364d42cc (127.9.176.196) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false replace: true } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { promote: false } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: false } } }
I20240502 01:55:31.151293 26194 raft_consensus.cc:2932] T 75d71cdcf4d44353acb7d2f18696977b P 6742b9db50c64ea394c2ca38ddee0872 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 30921110704f47f88d68a7d9364d42cc (127.9.176.196) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false replace: true } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { promote: false } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: false } } }
I20240502 01:55:31.152581 24726 catalog_manager.cc:5484] T 75d71cdcf4d44353acb7d2f18696977b P 96e5c2fd5d624c2da735e72b387a12d9 reported cstate change: config changed from index 10 to 11, 30921110704f47f88d68a7d9364d42cc (127.9.176.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "96e5c2fd5d624c2da735e72b387a12d9" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6742b9db50c64ea394c2ca38ddee0872" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 43479 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:55:31.164664 24867 consensus_queue.cc:237] T 75d71cdcf4d44353acb7d2f18696977b P 96e5c2fd5d624c2da735e72b387a12d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { promote: false } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: false } }
I20240502 01:55:31.166968 25933 raft_consensus.cc:1261] T 75d71cdcf4d44353acb7d2f18696977b P 30921110704f47f88d68a7d9364d42cc [term 1 FOLLOWER]: Refusing update from remote peer 96e5c2fd5d624c2da735e72b387a12d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 01:55:31.167625 25002 raft_consensus.cc:1261] T 75d71cdcf4d44353acb7d2f18696977b P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 1 FOLLOWER]: Refusing update from remote peer 96e5c2fd5d624c2da735e72b387a12d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 01:55:31.167608 27437 consensus_queue.cc:1035] T 75d71cdcf4d44353acb7d2f18696977b P 96e5c2fd5d624c2da735e72b387a12d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 01:55:31.168202 27421 consensus_queue.cc:1035] T 75d71cdcf4d44353acb7d2f18696977b P 96e5c2fd5d624c2da735e72b387a12d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 01:55:31.170233 27437 raft_consensus.cc:2932] T 75d71cdcf4d44353acb7d2f18696977b P 96e5c2fd5d624c2da735e72b387a12d9 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 6742b9db50c64ea394c2ca38ddee0872 (127.9.176.197) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { promote: false } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: false } } }
I20240502 01:55:31.170977 25933 raft_consensus.cc:2932] T 75d71cdcf4d44353acb7d2f18696977b P 30921110704f47f88d68a7d9364d42cc [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 6742b9db50c64ea394c2ca38ddee0872 (127.9.176.197) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { promote: false } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: false } } }
I20240502 01:55:31.171368 25002 raft_consensus.cc:2932] T 75d71cdcf4d44353acb7d2f18696977b P 75bd3f0e8d7c4e8aa92a353301afeb21 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 6742b9db50c64ea394c2ca38ddee0872 (127.9.176.197) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { promote: false } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: false } } }
I20240502 01:55:31.173840 24712 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 75d71cdcf4d44353acb7d2f18696977b with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:55:31.175129 24725 catalog_manager.cc:5484] T 75d71cdcf4d44353acb7d2f18696977b P 96e5c2fd5d624c2da735e72b387a12d9 reported cstate change: config changed from index 11 to 12, VOTER 6742b9db50c64ea394c2ca38ddee0872 (127.9.176.197) evicted. New cstate: current_term: 1 leader_uuid: "96e5c2fd5d624c2da735e72b387a12d9" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "96e5c2fd5d624c2da735e72b387a12d9" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 32783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "75bd3f0e8d7c4e8aa92a353301afeb21" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 41625 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30921110704f47f88d68a7d9364d42cc" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 39355 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:55:31.180512 26177 tablet_service.cc:1514] Processing DeleteTablet for tablet 75d71cdcf4d44353acb7d2f18696977b with delete_type TABLET_DATA_TOMBSTONED (TS 6742b9db50c64ea394c2ca38ddee0872 not found in new config with opid_index 12) from {username='slave'} at 127.0.0.1:43780
I20240502 01:55:31.181907 27450 tablet_replica.cc:330] T 75d71cdcf4d44353acb7d2f18696977b P 6742b9db50c64ea394c2ca38ddee0872: stopping tablet replica
I20240502 01:55:31.182178 27450 raft_consensus.cc:2229] T 75d71cdcf4d44353acb7d2f18696977b P 6742b9db50c64ea394c2ca38ddee0872 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:31.182572 27450 raft_consensus.cc:2258] T 75d71cdcf4d44353acb7d2f18696977b P 6742b9db50c64ea394c2ca38ddee0872 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:31.183874 27450 ts_tablet_manager.cc:1875] T 75d71cdcf4d44353acb7d2f18696977b P 6742b9db50c64ea394c2ca38ddee0872: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:31.188863 27450 ts_tablet_manager.cc:1888] T 75d71cdcf4d44353acb7d2f18696977b P 6742b9db50c64ea394c2ca38ddee0872: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.11
I20240502 01:55:31.188987 27450 log.cc:1196] T 75d71cdcf4d44353acb7d2f18696977b P 6742b9db50c64ea394c2ca38ddee0872: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/wal/wals/75d71cdcf4d44353acb7d2f18696977b
I20240502 01:55:31.189831 24711 catalog_manager.cc:4830] TS 6742b9db50c64ea394c2ca38ddee0872 (127.9.176.197:43479): tablet 75d71cdcf4d44353acb7d2f18696977b (table rebalance_test_table_1 [id=4e02aab5efa847ab8735d911dc7a4f5a]) successfully deleted
I20240502 01:55:31.361014 25914 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:31.371261 26177 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:31.371320 25112 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:55:31.380800 24982 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:55:31.392896 26045 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:31.418417 24848 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:55:31.576135 25914 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:31.593327 26177 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:31.609974 24848 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:55:31.624589 24982 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:55:31.625167 25112 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:55:31.634690 26045 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:31.867033  9923 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu pid 24787
I20240502 01:55:31.874975 24909 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:55:32.031847  9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu with pid 24787
I20240502 01:55:32.061519  9923 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu pid 24918
W20240502 01:55:32.061553 25068 connection.cc:535] client connection to 127.9.176.193:32783 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:55:32.061789 25068 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:55:32.063964 25068 consensus_peers.cc:485] T ce37ee2e946b4075b78f3c9456939848 P 672a55e7d4874c5d99f3c5e3d9550f15 -> Peer 96e5c2fd5d624c2da735e72b387a12d9 (127.9.176.193:32783): Couldn't send request to peer 96e5c2fd5d624c2da735e72b387a12d9. Status: Network error: Client connection negotiation failed: client connection to 127.9.176.193:32783: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:32.068291 25041 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:55:32.226476  9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu with pid 24918
W20240502 01:55:32.254098 26130 connection.cc:535] server connection from 127.9.176.194:60885 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:55:32.254729 25068 connection.cc:535] client connection to 127.9.176.194:41625 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:55:32.254838 24712 connection.cc:535] server connection from 127.9.176.194:56133 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 01:55:32.254987  9923 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu pid 25051
W20240502 01:55:32.256035 25068 consensus_peers.cc:485] T 450538b8217d40ce945d51e2c171ee36 P 672a55e7d4874c5d99f3c5e3d9550f15 -> Peer 75bd3f0e8d7c4e8aa92a353301afeb21 (127.9.176.194:41625): Couldn't send request to peer 75bd3f0e8d7c4e8aa92a353301afeb21. Status: Network error: Client connection negotiation failed: client connection to 127.9.176.194:41625: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:32.261078 25173 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:55:32.419842  9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu with pid 25051
W20240502 01:55:32.449057 25999 connection.cc:535] server connection from 127.9.176.195:34923 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:55:32.449371 24711 connection.cc:535] server connection from 127.9.176.195:46847 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 01:55:32.449717  9923 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu pid 25851
I20240502 01:55:32.456497 25973 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:55:32.579996  9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu with pid 25851
I20240502 01:55:32.603152  9923 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu pid 26115
I20240502 01:55:32.609720 26237 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:55:32.736380  9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu with pid 26115
I20240502 01:55:32.759390  9923 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu pid 25983
I20240502 01:55:32.767271 26107 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:55:32.890445  9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu with pid 25983
I20240502 01:55:32.913172  9923 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu pid 24696
I20240502 01:55:32.914398 24757 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:55:33.074432  9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu with pid 24696
2024-05-02T01:55:33Z chronyd exiting
[       OK ] LocationAwareRebalancingParamTest.Rebalance/idx3_rf3_t3_l_3_3 (22894 ms)
[ RUN      ] LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer/idx2_rf3_t3_l_3_2
I20240502 01:55:33.174238  9923 test_util.cc:268] Using random seed: -435898421
I20240502 01:55:33.176450  9923 ts_itest-base.cc:112] Starting cluster with:
I20240502 01:55:33.176519  9923 ts_itest-base.cc:113] --------------
I20240502 01:55:33.176542  9923 ts_itest-base.cc:114] 5 tablet servers
I20240502 01:55:33.176564  9923 ts_itest-base.cc:115] 3 replicas per TS
I20240502 01:55:33.176589  9923 ts_itest-base.cc:116] --------------
2024-05-02T01:55:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T01:55:33Z Disabled control of system clock
I20240502 01:55:33.233106  9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
/tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:41377
--webserver_interface=127.9.176.254
--webserver_port=0
--builtin_ntp_servers=127.9.176.212:46301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.176.254:41377
--location_mapping_cmd=/tmp/dist-test-taskJZxTxO/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/location-assignment.state --map /A:3 --map /B:2
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false
--master_client_location_assignment_enabled=false
--allow_unsafe_replication_factor with env {}
W20240502 01:55:33.589716 27543 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240502 01:55:33.590148 27543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:55:33.590257 27543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:55:33.590374 27543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:55:33.598287 27543 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:55:33.598464 27543 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 01:55:33.598587 27543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:55:33.598657 27543 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 01:55:33.598750 27543 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 01:55:33.608395 27543 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:46301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--location_mapping_cmd=/tmp/dist-test-taskJZxTxO/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/location-assignment.state --map /A:3 --map /B:2
--ipki_ca_key_size=768
--master_addresses=127.9.176.254:41377
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:41377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.9.176.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true

Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:55:33.609835 27543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:55:33.611738 27543 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:55:33.623734 27548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:33.624110 27551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:33.624507 27549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:55:33.625201 27543 server_base.cc:1034] running on GCE node
I20240502 01:55:33.625993 27543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:55:33.627447 27543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:55:33.628691 27543 hybrid_clock.cc:648] HybridClock initialized: now 1714614933628625 us; error 38 us; skew 500 ppm
I20240502 01:55:33.629015 27543 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:55:33.631398 27543 webserver.cc:460] Webserver started at http://127.9.176.254:33913/ using document root <none> and password file <none>
I20240502 01:55:33.632236 27543 fs_manager.cc:359] Metadata directory not provided
I20240502 01:55:33.632369 27543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:55:33.632692 27543 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:55:33.637552 27543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "65bfb7e9f0dd4d44be0039ee47dcc689"
format_stamp: "Formatted at 2024-05-02 01:55:33 on dist-test-slave-j3hh"
I20240502 01:55:33.638383 27543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "65bfb7e9f0dd4d44be0039ee47dcc689"
format_stamp: "Formatted at 2024-05-02 01:55:33 on dist-test-slave-j3hh"
I20240502 01:55:33.644026 27543 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.004s	sys 0.003s
I20240502 01:55:33.647701 27557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:55:33.649103 27543 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20240502 01:55:33.649305 27543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/wal
uuid: "65bfb7e9f0dd4d44be0039ee47dcc689"
format_stamp: "Formatted at 2024-05-02 01:55:33 on dist-test-slave-j3hh"
I20240502 01:55:33.649513 27543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:55:33.672487 27543 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:55:33.673426 27543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:55:33.673768 27543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:55:33.695188 27543 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.254:41377
I20240502 01:55:33.695212 27608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.254:41377 every 8 connection(s)
I20240502 01:55:33.696858 27543 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 01:55:33.697774  9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu as pid 27543
I20240502 01:55:33.698035  9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 01:55:33.702108 27609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:33.714897 27609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65bfb7e9f0dd4d44be0039ee47dcc689: Bootstrap starting.
I20240502 01:55:33.718230 27609 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65bfb7e9f0dd4d44be0039ee47dcc689: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:33.719290 27609 log.cc:824] T 00000000000000000000000000000000 P 65bfb7e9f0dd4d44be0039ee47dcc689: Log is configured to *not* fsync() on all Append() calls
I20240502 01:55:33.721872 27609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65bfb7e9f0dd4d44be0039ee47dcc689: No bootstrap required, opened a new log
I20240502 01:55:33.728605 27609 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65bfb7e9f0dd4d44be0039ee47dcc689 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65bfb7e9f0dd4d44be0039ee47dcc689" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 41377 } }
I20240502 01:55:33.729023 27609 raft_consensus.cc:374] T 00000000000000000000000000000000 P 65bfb7e9f0dd4d44be0039ee47dcc689 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:33.729142 27609 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65bfb7e9f0dd4d44be0039ee47dcc689 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65bfb7e9f0dd4d44be0039ee47dcc689, State: Initialized, Role: FOLLOWER
I20240502 01:55:33.729872 27609 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65bfb7e9f0dd4d44be0039ee47dcc689 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65bfb7e9f0dd4d44be0039ee47dcc689" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 41377 } }
I20240502 01:55:33.730104 27609 raft_consensus.cc:388] T 00000000000000000000000000000000 P 65bfb7e9f0dd4d44be0039ee47dcc689 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:55:33.730252 27609 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65bfb7e9f0dd4d44be0039ee47dcc689 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:55:33.730409 27609 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65bfb7e9f0dd4d44be0039ee47dcc689 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:33.732853 27609 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65bfb7e9f0dd4d44be0039ee47dcc689 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65bfb7e9f0dd4d44be0039ee47dcc689" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 41377 } }
I20240502 01:55:33.733362 27609 leader_election.cc:302] T 00000000000000000000000000000000 P 65bfb7e9f0dd4d44be0039ee47dcc689 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 65bfb7e9f0dd4d44be0039ee47dcc689; no voters: 
I20240502 01:55:33.733981 27609 leader_election.cc:288] T 00000000000000000000000000000000 P 65bfb7e9f0dd4d44be0039ee47dcc689 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240502 01:55:33.734125 27614 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 65bfb7e9f0dd4d44be0039ee47dcc689 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:55:33.734899 27614 raft_consensus.cc:686] T 00000000000000000000000000000000 P 65bfb7e9f0dd4d44be0039ee47dcc689 [term 1 LEADER]: Becoming Leader. State: Replica: 65bfb7e9f0dd4d44be0039ee47dcc689, State: Running, Role: LEADER
I20240502 01:55:33.735451 27614 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65bfb7e9f0dd4d44be0039ee47dcc689 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65bfb7e9f0dd4d44be0039ee47dcc689" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 41377 } }
I20240502 01:55:33.736238 27609 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65bfb7e9f0dd4d44be0039ee47dcc689 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 01:55:33.738710 27615 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65bfb7e9f0dd4d44be0039ee47dcc689 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "65bfb7e9f0dd4d44be0039ee47dcc689" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65bfb7e9f0dd4d44be0039ee47dcc689" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 41377 } } }
I20240502 01:55:33.738711 27616 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65bfb7e9f0dd4d44be0039ee47dcc689 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65bfb7e9f0dd4d44be0039ee47dcc689. Latest consensus state: current_term: 1 leader_uuid: "65bfb7e9f0dd4d44be0039ee47dcc689" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65bfb7e9f0dd4d44be0039ee47dcc689" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 41377 } } }
I20240502 01:55:33.739104 27615 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65bfb7e9f0dd4d44be0039ee47dcc689 [sys.catalog]: This master's current role is: LEADER
I20240502 01:55:33.739171 27616 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65bfb7e9f0dd4d44be0039ee47dcc689 [sys.catalog]: This master's current role is: LEADER
I20240502 01:55:33.741516 27619 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 01:55:33.748790 27619 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 01:55:33.760329 27619 catalog_manager.cc:1347] Generated new cluster ID: df9eb592ea9246a29742a7fc1576138a
I20240502 01:55:33.760485 27619 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 01:55:33.780193 27619 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 01:55:33.781564 27619 catalog_manager.cc:1504] Loading token signing keys...
I20240502 01:55:33.799583 27619 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 65bfb7e9f0dd4d44be0039ee47dcc689: Generated new TSK 0
I20240502 01:55:33.800844 27619 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 01:55:33.810953  9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
/tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.193:0
--local_ip_for_outbound_sockets=127.9.176.193
--webserver_interface=127.9.176.193
--webserver_port=0
--tserver_master_addrs=127.9.176.254:41377
--builtin_ntp_servers=127.9.176.212:46301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 01:55:34.184311 27634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:55:34.184710 27634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:55:34.184818 27634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:55:34.192761 27634 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:55:34.192910 27634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:55:34.193033 27634 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.193
I20240502 01:55:34.202829 27634 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:46301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.9.176.193
--webserver_port=0
--tserver_master_addrs=127.9.176.254:41377
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.193
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:55:34.204360 27634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:55:34.206246 27634 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:55:34.219521 27639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:34.220160 27640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:34.220599 27642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:55:34.221410 27634 server_base.cc:1034] running on GCE node
I20240502 01:55:34.222105 27634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:55:34.223174 27634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:55:34.224395 27634 hybrid_clock.cc:648] HybridClock initialized: now 1714614934224325 us; error 63 us; skew 500 ppm
I20240502 01:55:34.224743 27634 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:55:34.227268 27634 webserver.cc:460] Webserver started at http://127.9.176.193:37065/ using document root <none> and password file <none>
I20240502 01:55:34.228163 27634 fs_manager.cc:359] Metadata directory not provided
I20240502 01:55:34.228302 27634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:55:34.228647 27634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:55:34.231499 27634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "d50f07af87d74f039fb2b43cf166fa43"
format_stamp: "Formatted at 2024-05-02 01:55:34 on dist-test-slave-j3hh"
I20240502 01:55:34.232460 27634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "d50f07af87d74f039fb2b43cf166fa43"
format_stamp: "Formatted at 2024-05-02 01:55:34 on dist-test-slave-j3hh"
I20240502 01:55:34.238400 27634 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.005s	sys 0.000s
I20240502 01:55:34.242444 27648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:55:34.244071 27634 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240502 01:55:34.244284 27634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "d50f07af87d74f039fb2b43cf166fa43"
format_stamp: "Formatted at 2024-05-02 01:55:34 on dist-test-slave-j3hh"
I20240502 01:55:34.244540 27634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:55:34.273752 27634 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:55:34.274770 27634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:55:34.275086 27634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:55:34.276302 27634 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:55:34.278081 27634 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:55:34.278201 27634 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:55:34.278331 27634 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:55:34.278409 27634 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:55:34.314095 27634 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.193:34203
I20240502 01:55:34.314150 27761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.193:34203 every 8 connection(s)
I20240502 01:55:34.315713 27634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 01:55:34.316784  9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu as pid 27634
I20240502 01:55:34.316999  9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 01:55:34.320994  9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
/tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.194:0
--local_ip_for_outbound_sockets=127.9.176.194
--webserver_interface=127.9.176.194
--webserver_port=0
--tserver_master_addrs=127.9.176.254:41377
--builtin_ntp_servers=127.9.176.212:46301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:55:34.330159 27762 heartbeater.cc:347] Connected to a master server at 127.9.176.254:41377
I20240502 01:55:34.330515 27762 heartbeater.cc:464] Registering TS with master...
I20240502 01:55:34.331432 27762 heartbeater.cc:511] Master 127.9.176.254:41377 requested a full tablet report, sending...
I20240502 01:55:34.397516 27574 ts_manager.cc:194] Registered new tserver with Master: d50f07af87d74f039fb2b43cf166fa43 (127.9.176.193:34203)
I20240502 01:55:34.399456 27574 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.176.193:36571
W20240502 01:55:34.716374 27766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:55:34.716810 27766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:55:34.716989 27766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:55:34.725898 27766 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:55:34.726035 27766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:55:34.726153 27766 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.194
I20240502 01:55:34.735980 27766 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:46301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.9.176.194
--webserver_port=0
--tserver_master_addrs=127.9.176.254:41377
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.194
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:55:34.737368 27766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:55:34.739130 27766 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:55:34.751593 27772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:34.752022 27773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:34.752626 27775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:55:34.753451 27766 server_base.cc:1034] running on GCE node
I20240502 01:55:34.754106 27766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:55:34.755132 27766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:55:34.756340 27766 hybrid_clock.cc:648] HybridClock initialized: now 1714614934756291 us; error 38 us; skew 500 ppm
I20240502 01:55:34.756673 27766 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:55:34.759081 27766 webserver.cc:460] Webserver started at http://127.9.176.194:39297/ using document root <none> and password file <none>
I20240502 01:55:34.760159 27766 fs_manager.cc:359] Metadata directory not provided
I20240502 01:55:34.760308 27766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:55:34.760748 27766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:55:34.764822 27766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "a64152fd9595478ca767a9f248ef3b85"
format_stamp: "Formatted at 2024-05-02 01:55:34 on dist-test-slave-j3hh"
I20240502 01:55:34.766111 27766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "a64152fd9595478ca767a9f248ef3b85"
format_stamp: "Formatted at 2024-05-02 01:55:34 on dist-test-slave-j3hh"
I20240502 01:55:34.772867 27766 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.006s	sys 0.000s
I20240502 01:55:34.776805 27781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:55:34.778282 27766 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.001s
I20240502 01:55:34.778483 27766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "a64152fd9595478ca767a9f248ef3b85"
format_stamp: "Formatted at 2024-05-02 01:55:34 on dist-test-slave-j3hh"
I20240502 01:55:34.778719 27766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:55:34.804859 27766 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:55:34.805843 27766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:55:34.806146 27766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:55:34.807240 27766 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:55:34.808871 27766 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:55:34.808982 27766 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:55:34.809103 27766 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:55:34.809175 27766 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:55:34.843129 27766 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.194:38689
I20240502 01:55:34.843179 27894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.194:38689 every 8 connection(s)
I20240502 01:55:34.844475 27766 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 01:55:34.849908  9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu as pid 27766
I20240502 01:55:34.850147  9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 01:55:34.853852  9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
/tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.195:0
--local_ip_for_outbound_sockets=127.9.176.195
--webserver_interface=127.9.176.195
--webserver_port=0
--tserver_master_addrs=127.9.176.254:41377
--builtin_ntp_servers=127.9.176.212:46301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:55:34.859318 27895 heartbeater.cc:347] Connected to a master server at 127.9.176.254:41377
I20240502 01:55:34.859714 27895 heartbeater.cc:464] Registering TS with master...
I20240502 01:55:34.860762 27895 heartbeater.cc:511] Master 127.9.176.254:41377 requested a full tablet report, sending...
I20240502 01:55:34.910548 27574 ts_manager.cc:194] Registered new tserver with Master: a64152fd9595478ca767a9f248ef3b85 (127.9.176.194:38689)
I20240502 01:55:34.911999 27574 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.176.194:48601
W20240502 01:55:35.225114 27899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:55:35.225425 27899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:55:35.225528 27899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:55:35.233111 27899 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:55:35.233239 27899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:55:35.233356 27899 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.195
I20240502 01:55:35.242645 27899 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:46301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.9.176.195
--webserver_port=0
--tserver_master_addrs=127.9.176.254:41377
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.195
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:55:35.243940 27899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:55:35.245714 27899 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:55:35.259130 27905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:35.259779 27906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:35.261893 27908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:55:35.262809 27899 server_base.cc:1034] running on GCE node
I20240502 01:55:35.263679 27899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:55:35.264822 27899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:55:35.266067 27899 hybrid_clock.cc:648] HybridClock initialized: now 1714614935266020 us; error 42 us; skew 500 ppm
I20240502 01:55:35.266439 27899 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:55:35.269281 27899 webserver.cc:460] Webserver started at http://127.9.176.195:43933/ using document root <none> and password file <none>
I20240502 01:55:35.270179 27899 fs_manager.cc:359] Metadata directory not provided
I20240502 01:55:35.270332 27899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:55:35.270704 27899 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:55:35.273850 27899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "b550bed5a64f466c8ba5a1761f3bef39"
format_stamp: "Formatted at 2024-05-02 01:55:35 on dist-test-slave-j3hh"
I20240502 01:55:35.274847 27899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "b550bed5a64f466c8ba5a1761f3bef39"
format_stamp: "Formatted at 2024-05-02 01:55:35 on dist-test-slave-j3hh"
I20240502 01:55:35.281415 27899 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.000s	sys 0.008s
I20240502 01:55:35.285796 27914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:55:35.287602 27899 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.001s	sys 0.000s
I20240502 01:55:35.287842 27899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "b550bed5a64f466c8ba5a1761f3bef39"
format_stamp: "Formatted at 2024-05-02 01:55:35 on dist-test-slave-j3hh"
I20240502 01:55:35.288106 27899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:55:35.305498 27899 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:55:35.306499 27899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:55:35.306808 27899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:55:35.307909 27899 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:55:35.309585 27899 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:55:35.309695 27899 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:55:35.309831 27899 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:55:35.309909 27899 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:55:35.343510 27899 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.195:40013
I20240502 01:55:35.343588 28028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.195:40013 every 8 connection(s)
I20240502 01:55:35.344902 27899 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 01:55:35.353869  9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu as pid 27899
I20240502 01:55:35.354108  9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 01:55:35.358000  9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
/tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.196:0
--local_ip_for_outbound_sockets=127.9.176.196
--webserver_interface=127.9.176.196
--webserver_port=0
--tserver_master_addrs=127.9.176.254:41377
--builtin_ntp_servers=127.9.176.212:46301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:55:35.359156 28029 heartbeater.cc:347] Connected to a master server at 127.9.176.254:41377
I20240502 01:55:35.359483 28029 heartbeater.cc:464] Registering TS with master...
I20240502 01:55:35.360428 28029 heartbeater.cc:511] Master 127.9.176.254:41377 requested a full tablet report, sending...
I20240502 01:55:35.404599 27762 heartbeater.cc:503] Master 127.9.176.254:41377 was elected leader, sending a full tablet report...
I20240502 01:55:35.410604 27574 ts_manager.cc:194] Registered new tserver with Master: b550bed5a64f466c8ba5a1761f3bef39 (127.9.176.195:40013)
I20240502 01:55:35.411633 27574 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.176.195:52549
W20240502 01:55:35.726799 28033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:55:35.727140 28033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:55:35.727283 28033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:55:35.735304 28033 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:55:35.735450 28033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:55:35.735678 28033 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.196
I20240502 01:55:35.745363 28033 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:46301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.9.176.196
--webserver_port=0
--tserver_master_addrs=127.9.176.254:41377
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.196
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:55:35.746747 28033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:55:35.748560 28033 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:55:35.761999 28038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:35.762460 28039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:35.762993 28041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:55:35.763805 28033 server_base.cc:1034] running on GCE node
I20240502 01:55:35.764459 28033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:55:35.765494 28033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:55:35.766717 28033 hybrid_clock.cc:648] HybridClock initialized: now 1714614935766665 us; error 45 us; skew 500 ppm
I20240502 01:55:35.767086 28033 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:55:35.769744 28033 webserver.cc:460] Webserver started at http://127.9.176.196:37483/ using document root <none> and password file <none>
I20240502 01:55:35.770615 28033 fs_manager.cc:359] Metadata directory not provided
I20240502 01:55:35.770751 28033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:55:35.771124 28033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:55:35.774117 28033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "543169fd701141c181c55566d2cae00d"
format_stamp: "Formatted at 2024-05-02 01:55:35 on dist-test-slave-j3hh"
I20240502 01:55:35.775097 28033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "543169fd701141c181c55566d2cae00d"
format_stamp: "Formatted at 2024-05-02 01:55:35 on dist-test-slave-j3hh"
I20240502 01:55:35.781379 28033 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.008s	sys 0.000s
I20240502 01:55:35.785861 28047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:55:35.787550 28033 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.002s	sys 0.000s
I20240502 01:55:35.787817 28033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "543169fd701141c181c55566d2cae00d"
format_stamp: "Formatted at 2024-05-02 01:55:35 on dist-test-slave-j3hh"
I20240502 01:55:35.788070 28033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:55:35.816259 28033 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:55:35.817261 28033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:55:35.817659 28033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:55:35.819070 28033 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:55:35.820843 28033 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:55:35.820946 28033 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:55:35.821079 28033 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:55:35.821158 28033 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:55:35.856465 28033 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.196:38773
I20240502 01:55:35.856530 28159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.196:38773 every 8 connection(s)
I20240502 01:55:35.857877 28033 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 01:55:35.865439  9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu as pid 28033
I20240502 01:55:35.865655  9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 01:55:35.869140  9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
/tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.197:0
--local_ip_for_outbound_sockets=127.9.176.197
--webserver_interface=127.9.176.197
--webserver_port=0
--tserver_master_addrs=127.9.176.254:41377
--builtin_ntp_servers=127.9.176.212:46301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:55:35.873172 28160 heartbeater.cc:347] Connected to a master server at 127.9.176.254:41377
I20240502 01:55:35.873540 28160 heartbeater.cc:464] Registering TS with master...
I20240502 01:55:35.874615 28160 heartbeater.cc:511] Master 127.9.176.254:41377 requested a full tablet report, sending...
I20240502 01:55:35.915100 27895 heartbeater.cc:503] Master 127.9.176.254:41377 was elected leader, sending a full tablet report...
I20240502 01:55:35.944022 27574 ts_manager.cc:194] Registered new tserver with Master: 543169fd701141c181c55566d2cae00d (127.9.176.196:38773)
I20240502 01:55:35.944975 27574 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.176.196:55101
W20240502 01:55:36.251852 28164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:55:36.252190 28164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:55:36.252355 28164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:55:36.260385 28164 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:55:36.260532 28164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:55:36.260701 28164 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.197
I20240502 01:55:36.270295 28164 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:46301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.197:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.9.176.197
--webserver_port=0
--tserver_master_addrs=127.9.176.254:41377
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.197
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:55:36.271811 28164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:55:36.273739 28164 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:55:36.287547 28170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:36.287967 28173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:36.288415 28171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:55:36.289268 28164 server_base.cc:1034] running on GCE node
I20240502 01:55:36.290000 28164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:55:36.291083 28164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:55:36.292310 28164 hybrid_clock.cc:648] HybridClock initialized: now 1714614936292247 us; error 55 us; skew 500 ppm
I20240502 01:55:36.292672 28164 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:55:36.295531 28164 webserver.cc:460] Webserver started at http://127.9.176.197:42969/ using document root <none> and password file <none>
I20240502 01:55:36.296597 28164 fs_manager.cc:359] Metadata directory not provided
I20240502 01:55:36.296737 28164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:55:36.297058 28164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:55:36.300257 28164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "c541378ca18944c49c37f49097bc7f81"
format_stamp: "Formatted at 2024-05-02 01:55:36 on dist-test-slave-j3hh"
I20240502 01:55:36.301218 28164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "c541378ca18944c49c37f49097bc7f81"
format_stamp: "Formatted at 2024-05-02 01:55:36 on dist-test-slave-j3hh"
I20240502 01:55:36.307281 28164 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.006s	sys 0.001s
I20240502 01:55:36.311133 28179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:55:36.312644 28164 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240502 01:55:36.312848 28164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "c541378ca18944c49c37f49097bc7f81"
format_stamp: "Formatted at 2024-05-02 01:55:36 on dist-test-slave-j3hh"
I20240502 01:55:36.313082 28164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:55:36.359894 28164 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:55:36.360790 28164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:55:36.361061 28164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:55:36.362105 28164 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:55:36.363768 28164 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:55:36.363862 28164 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:55:36.363991 28164 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:55:36.364068 28164 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:55:36.399902 28164 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.197:32773
I20240502 01:55:36.399963 28293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.197:32773 every 8 connection(s)
I20240502 01:55:36.401376 28164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 01:55:36.404790  9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu as pid 28164
I20240502 01:55:36.404989  9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 01:55:36.414631 28029 heartbeater.cc:503] Master 127.9.176.254:41377 was elected leader, sending a full tablet report...
I20240502 01:55:36.415122 28294 heartbeater.cc:347] Connected to a master server at 127.9.176.254:41377
I20240502 01:55:36.415416 28294 heartbeater.cc:464] Registering TS with master...
I20240502 01:55:36.416337 28294 heartbeater.cc:511] Master 127.9.176.254:41377 requested a full tablet report, sending...
I20240502 01:55:36.466008 27573 ts_manager.cc:194] Registered new tserver with Master: c541378ca18944c49c37f49097bc7f81 (127.9.176.197:32773)
I20240502 01:55:36.467115 27573 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.176.197:50263
I20240502 01:55:36.479117  9923 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240502 01:55:36.499537  9923 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240502 01:55:36.499763  9923 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu pid 28164
I20240502 01:55:36.508615 28289 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:55:36.593123  9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu with pid 28164
I20240502 01:55:36.611073  9923 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu pid 28033
I20240502 01:55:36.619784 28155 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:55:36.703966  9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu with pid 28033
I20240502 01:55:40.474011 27573 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41110:
name: "rebalance_test_table_0"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
W20240502 01:55:40.481923 27573 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 01:55:40.521139 27829 tablet_service.cc:1467] Processing CreateTablet for tablet f1597c56a1a74b37bb5e18e8c4662f46 (DEFAULT_TABLE table=rebalance_test_table_0 [id=8ef52b859d42498b92b6b2ec0805663b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:40.521726 27696 tablet_service.cc:1467] Processing CreateTablet for tablet f1597c56a1a74b37bb5e18e8c4662f46 (DEFAULT_TABLE table=rebalance_test_table_0 [id=8ef52b859d42498b92b6b2ec0805663b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:40.522485 27829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1597c56a1a74b37bb5e18e8c4662f46. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:40.523013 27827 tablet_service.cc:1467] Processing CreateTablet for tablet 8a29996e3d594ebebae8b5fe7e02aa8c (DEFAULT_TABLE table=rebalance_test_table_0 [id=8ef52b859d42498b92b6b2ec0805663b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:40.523545 27828 tablet_service.cc:1467] Processing CreateTablet for tablet 9a16d465aa2b426484868a2b212c5d00 (DEFAULT_TABLE table=rebalance_test_table_0 [id=8ef52b859d42498b92b6b2ec0805663b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:40.524248 27694 tablet_service.cc:1467] Processing CreateTablet for tablet 8a29996e3d594ebebae8b5fe7e02aa8c (DEFAULT_TABLE table=rebalance_test_table_0 [id=8ef52b859d42498b92b6b2ec0805663b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:40.524577 27827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a29996e3d594ebebae8b5fe7e02aa8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:40.524124 27962 tablet_service.cc:1467] Processing CreateTablet for tablet f1597c56a1a74b37bb5e18e8c4662f46 (DEFAULT_TABLE table=rebalance_test_table_0 [id=8ef52b859d42498b92b6b2ec0805663b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:40.525074 27695 tablet_service.cc:1467] Processing CreateTablet for tablet 9a16d465aa2b426484868a2b212c5d00 (DEFAULT_TABLE table=rebalance_test_table_0 [id=8ef52b859d42498b92b6b2ec0805663b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:40.525650 27962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1597c56a1a74b37bb5e18e8c4662f46. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:40.526252 27828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a16d465aa2b426484868a2b212c5d00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:40.526212 27960 tablet_service.cc:1467] Processing CreateTablet for tablet 8a29996e3d594ebebae8b5fe7e02aa8c (DEFAULT_TABLE table=rebalance_test_table_0 [id=8ef52b859d42498b92b6b2ec0805663b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:40.526759 27960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a29996e3d594ebebae8b5fe7e02aa8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:40.527707 27961 tablet_service.cc:1467] Processing CreateTablet for tablet 9a16d465aa2b426484868a2b212c5d00 (DEFAULT_TABLE table=rebalance_test_table_0 [id=8ef52b859d42498b92b6b2ec0805663b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:40.528261 27961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a16d465aa2b426484868a2b212c5d00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:40.529191 27696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1597c56a1a74b37bb5e18e8c4662f46. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:40.529687 27694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a29996e3d594ebebae8b5fe7e02aa8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:40.531847 27695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a16d465aa2b426484868a2b212c5d00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:40.558485 28319 tablet_bootstrap.cc:492] T 8a29996e3d594ebebae8b5fe7e02aa8c P a64152fd9595478ca767a9f248ef3b85: Bootstrap starting.
I20240502 01:55:40.560567 28320 tablet_bootstrap.cc:492] T 9a16d465aa2b426484868a2b212c5d00 P d50f07af87d74f039fb2b43cf166fa43: Bootstrap starting.
I20240502 01:55:40.560818 28321 tablet_bootstrap.cc:492] T f1597c56a1a74b37bb5e18e8c4662f46 P b550bed5a64f466c8ba5a1761f3bef39: Bootstrap starting.
I20240502 01:55:40.562911 28319 tablet_bootstrap.cc:654] T 8a29996e3d594ebebae8b5fe7e02aa8c P a64152fd9595478ca767a9f248ef3b85: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:40.563828 28321 tablet_bootstrap.cc:654] T f1597c56a1a74b37bb5e18e8c4662f46 P b550bed5a64f466c8ba5a1761f3bef39: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:40.564142 28320 tablet_bootstrap.cc:654] T 9a16d465aa2b426484868a2b212c5d00 P d50f07af87d74f039fb2b43cf166fa43: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:40.564400 28319 log.cc:824] T 8a29996e3d594ebebae8b5fe7e02aa8c P a64152fd9595478ca767a9f248ef3b85: Log is configured to *not* fsync() on all Append() calls
I20240502 01:55:40.565619 28320 log.cc:824] T 9a16d465aa2b426484868a2b212c5d00 P d50f07af87d74f039fb2b43cf166fa43: Log is configured to *not* fsync() on all Append() calls
I20240502 01:55:40.565761 28321 log.cc:824] T f1597c56a1a74b37bb5e18e8c4662f46 P b550bed5a64f466c8ba5a1761f3bef39: Log is configured to *not* fsync() on all Append() calls
I20240502 01:55:40.568563 28320 tablet_bootstrap.cc:492] T 9a16d465aa2b426484868a2b212c5d00 P d50f07af87d74f039fb2b43cf166fa43: No bootstrap required, opened a new log
I20240502 01:55:40.568580 28319 tablet_bootstrap.cc:492] T 8a29996e3d594ebebae8b5fe7e02aa8c P a64152fd9595478ca767a9f248ef3b85: No bootstrap required, opened a new log
I20240502 01:55:40.568874 28320 ts_tablet_manager.cc:1367] T 9a16d465aa2b426484868a2b212c5d00 P d50f07af87d74f039fb2b43cf166fa43: Time spent bootstrapping tablet: real 0.009s	user 0.006s	sys 0.001s
I20240502 01:55:40.568905 28319 ts_tablet_manager.cc:1367] T 8a29996e3d594ebebae8b5fe7e02aa8c P a64152fd9595478ca767a9f248ef3b85: Time spent bootstrapping tablet: real 0.011s	user 0.009s	sys 0.000s
I20240502 01:55:40.570662 28321 tablet_bootstrap.cc:492] T f1597c56a1a74b37bb5e18e8c4662f46 P b550bed5a64f466c8ba5a1761f3bef39: No bootstrap required, opened a new log
I20240502 01:55:40.571038 28321 ts_tablet_manager.cc:1367] T f1597c56a1a74b37bb5e18e8c4662f46 P b550bed5a64f466c8ba5a1761f3bef39: Time spent bootstrapping tablet: real 0.010s	user 0.003s	sys 0.004s
I20240502 01:55:40.576049 28320 raft_consensus.cc:348] T 9a16d465aa2b426484868a2b212c5d00 P d50f07af87d74f039fb2b43cf166fa43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } }
I20240502 01:55:40.576539 28320 raft_consensus.cc:374] T 9a16d465aa2b426484868a2b212c5d00 P d50f07af87d74f039fb2b43cf166fa43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:40.576678 28320 raft_consensus.cc:729] T 9a16d465aa2b426484868a2b212c5d00 P d50f07af87d74f039fb2b43cf166fa43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d50f07af87d74f039fb2b43cf166fa43, State: Initialized, Role: FOLLOWER
I20240502 01:55:40.577497 28320 consensus_queue.cc:260] T 9a16d465aa2b426484868a2b212c5d00 P d50f07af87d74f039fb2b43cf166fa43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } }
I20240502 01:55:40.580029 28319 raft_consensus.cc:348] T 8a29996e3d594ebebae8b5fe7e02aa8c P a64152fd9595478ca767a9f248ef3b85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } }
I20240502 01:55:40.580619 28319 raft_consensus.cc:374] T 8a29996e3d594ebebae8b5fe7e02aa8c P a64152fd9595478ca767a9f248ef3b85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:40.580744 28319 raft_consensus.cc:729] T 8a29996e3d594ebebae8b5fe7e02aa8c P a64152fd9595478ca767a9f248ef3b85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64152fd9595478ca767a9f248ef3b85, State: Initialized, Role: FOLLOWER
I20240502 01:55:40.581574 28319 consensus_queue.cc:260] T 8a29996e3d594ebebae8b5fe7e02aa8c P a64152fd9595478ca767a9f248ef3b85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } }
I20240502 01:55:40.582533 28320 ts_tablet_manager.cc:1398] T 9a16d465aa2b426484868a2b212c5d00 P d50f07af87d74f039fb2b43cf166fa43: Time spent starting tablet: real 0.014s	user 0.009s	sys 0.004s
I20240502 01:55:40.583777 28319 ts_tablet_manager.cc:1398] T 8a29996e3d594ebebae8b5fe7e02aa8c P a64152fd9595478ca767a9f248ef3b85: Time spent starting tablet: real 0.015s	user 0.016s	sys 0.000s
I20240502 01:55:40.584071 28320 tablet_bootstrap.cc:492] T 8a29996e3d594ebebae8b5fe7e02aa8c P d50f07af87d74f039fb2b43cf166fa43: Bootstrap starting.
I20240502 01:55:40.584444 28319 tablet_bootstrap.cc:492] T 9a16d465aa2b426484868a2b212c5d00 P a64152fd9595478ca767a9f248ef3b85: Bootstrap starting.
I20240502 01:55:40.585469 28321 raft_consensus.cc:348] T f1597c56a1a74b37bb5e18e8c4662f46 P b550bed5a64f466c8ba5a1761f3bef39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } }
I20240502 01:55:40.586097 28321 raft_consensus.cc:374] T f1597c56a1a74b37bb5e18e8c4662f46 P b550bed5a64f466c8ba5a1761f3bef39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:40.586258 28321 raft_consensus.cc:729] T f1597c56a1a74b37bb5e18e8c4662f46 P b550bed5a64f466c8ba5a1761f3bef39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b550bed5a64f466c8ba5a1761f3bef39, State: Initialized, Role: FOLLOWER
I20240502 01:55:40.586344 28319 tablet_bootstrap.cc:654] T 9a16d465aa2b426484868a2b212c5d00 P a64152fd9595478ca767a9f248ef3b85: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:40.587306 28321 consensus_queue.cc:260] T f1597c56a1a74b37bb5e18e8c4662f46 P b550bed5a64f466c8ba5a1761f3bef39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } }
I20240502 01:55:40.587822 28320 tablet_bootstrap.cc:654] T 8a29996e3d594ebebae8b5fe7e02aa8c P d50f07af87d74f039fb2b43cf166fa43: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:40.594872 28321 ts_tablet_manager.cc:1398] T f1597c56a1a74b37bb5e18e8c4662f46 P b550bed5a64f466c8ba5a1761f3bef39: Time spent starting tablet: real 0.024s	user 0.013s	sys 0.008s
I20240502 01:55:40.594594 28319 tablet_bootstrap.cc:492] T 9a16d465aa2b426484868a2b212c5d00 P a64152fd9595478ca767a9f248ef3b85: No bootstrap required, opened a new log
I20240502 01:55:40.595360 28319 ts_tablet_manager.cc:1367] T 9a16d465aa2b426484868a2b212c5d00 P a64152fd9595478ca767a9f248ef3b85: Time spent bootstrapping tablet: real 0.011s	user 0.006s	sys 0.004s
I20240502 01:55:40.595643 28321 tablet_bootstrap.cc:492] T 8a29996e3d594ebebae8b5fe7e02aa8c P b550bed5a64f466c8ba5a1761f3bef39: Bootstrap starting.
I20240502 01:55:40.597059 28319 raft_consensus.cc:348] T 9a16d465aa2b426484868a2b212c5d00 P a64152fd9595478ca767a9f248ef3b85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } }
I20240502 01:55:40.597436 28319 raft_consensus.cc:374] T 9a16d465aa2b426484868a2b212c5d00 P a64152fd9595478ca767a9f248ef3b85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:40.597569 28319 raft_consensus.cc:729] T 9a16d465aa2b426484868a2b212c5d00 P a64152fd9595478ca767a9f248ef3b85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64152fd9595478ca767a9f248ef3b85, State: Initialized, Role: FOLLOWER
I20240502 01:55:40.597921 28321 tablet_bootstrap.cc:654] T 8a29996e3d594ebebae8b5fe7e02aa8c P b550bed5a64f466c8ba5a1761f3bef39: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:40.597966 28319 consensus_queue.cc:260] T 9a16d465aa2b426484868a2b212c5d00 P a64152fd9595478ca767a9f248ef3b85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } }
I20240502 01:55:40.599900 28319 ts_tablet_manager.cc:1398] T 9a16d465aa2b426484868a2b212c5d00 P a64152fd9595478ca767a9f248ef3b85: Time spent starting tablet: real 0.004s	user 0.003s	sys 0.000s
I20240502 01:55:40.600360 28319 tablet_bootstrap.cc:492] T f1597c56a1a74b37bb5e18e8c4662f46 P a64152fd9595478ca767a9f248ef3b85: Bootstrap starting.
I20240502 01:55:40.600667 28320 tablet_bootstrap.cc:492] T 8a29996e3d594ebebae8b5fe7e02aa8c P d50f07af87d74f039fb2b43cf166fa43: No bootstrap required, opened a new log
I20240502 01:55:40.600870 28320 ts_tablet_manager.cc:1367] T 8a29996e3d594ebebae8b5fe7e02aa8c P d50f07af87d74f039fb2b43cf166fa43: Time spent bootstrapping tablet: real 0.017s	user 0.011s	sys 0.001s
W20240502 01:55:40.601245 27896 tablet.cc:2369] T 8a29996e3d594ebebae8b5fe7e02aa8c P a64152fd9595478ca767a9f248ef3b85: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:55:40.601909 28320 raft_consensus.cc:348] T 8a29996e3d594ebebae8b5fe7e02aa8c P d50f07af87d74f039fb2b43cf166fa43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } }
I20240502 01:55:40.602100 28320 raft_consensus.cc:374] T 8a29996e3d594ebebae8b5fe7e02aa8c P d50f07af87d74f039fb2b43cf166fa43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:40.602200 28320 raft_consensus.cc:729] T 8a29996e3d594ebebae8b5fe7e02aa8c P d50f07af87d74f039fb2b43cf166fa43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d50f07af87d74f039fb2b43cf166fa43, State: Initialized, Role: FOLLOWER
W20240502 01:55:40.602313 28030 tablet.cc:2369] T f1597c56a1a74b37bb5e18e8c4662f46 P b550bed5a64f466c8ba5a1761f3bef39: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:55:40.602475 28320 consensus_queue.cc:260] T 8a29996e3d594ebebae8b5fe7e02aa8c P d50f07af87d74f039fb2b43cf166fa43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } }
I20240502 01:55:40.602943 28320 ts_tablet_manager.cc:1398] T 8a29996e3d594ebebae8b5fe7e02aa8c P d50f07af87d74f039fb2b43cf166fa43: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 01:55:40.603276 28320 tablet_bootstrap.cc:492] T f1597c56a1a74b37bb5e18e8c4662f46 P d50f07af87d74f039fb2b43cf166fa43: Bootstrap starting.
I20240502 01:55:40.603682 28319 tablet_bootstrap.cc:654] T f1597c56a1a74b37bb5e18e8c4662f46 P a64152fd9595478ca767a9f248ef3b85: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:40.605139 28320 tablet_bootstrap.cc:654] T f1597c56a1a74b37bb5e18e8c4662f46 P d50f07af87d74f039fb2b43cf166fa43: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:40.606923 28319 tablet_bootstrap.cc:492] T f1597c56a1a74b37bb5e18e8c4662f46 P a64152fd9595478ca767a9f248ef3b85: No bootstrap required, opened a new log
I20240502 01:55:40.607126 28319 ts_tablet_manager.cc:1367] T f1597c56a1a74b37bb5e18e8c4662f46 P a64152fd9595478ca767a9f248ef3b85: Time spent bootstrapping tablet: real 0.007s	user 0.004s	sys 0.000s
I20240502 01:55:40.607915 28319 raft_consensus.cc:348] T f1597c56a1a74b37bb5e18e8c4662f46 P a64152fd9595478ca767a9f248ef3b85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } }
I20240502 01:55:40.608172 28319 raft_consensus.cc:374] T f1597c56a1a74b37bb5e18e8c4662f46 P a64152fd9595478ca767a9f248ef3b85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:40.608263 28319 raft_consensus.cc:729] T f1597c56a1a74b37bb5e18e8c4662f46 P a64152fd9595478ca767a9f248ef3b85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64152fd9595478ca767a9f248ef3b85, State: Initialized, Role: FOLLOWER
I20240502 01:55:40.608548 28319 consensus_queue.cc:260] T f1597c56a1a74b37bb5e18e8c4662f46 P a64152fd9595478ca767a9f248ef3b85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } }
I20240502 01:55:40.609066 28321 tablet_bootstrap.cc:492] T 8a29996e3d594ebebae8b5fe7e02aa8c P b550bed5a64f466c8ba5a1761f3bef39: No bootstrap required, opened a new log
I20240502 01:55:40.609133 28319 ts_tablet_manager.cc:1398] T f1597c56a1a74b37bb5e18e8c4662f46 P a64152fd9595478ca767a9f248ef3b85: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240502 01:55:40.609251 28321 ts_tablet_manager.cc:1367] T 8a29996e3d594ebebae8b5fe7e02aa8c P b550bed5a64f466c8ba5a1761f3bef39: Time spent bootstrapping tablet: real 0.014s	user 0.000s	sys 0.005s
I20240502 01:55:40.610229 28321 raft_consensus.cc:348] T 8a29996e3d594ebebae8b5fe7e02aa8c P b550bed5a64f466c8ba5a1761f3bef39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } }
I20240502 01:55:40.610478 28321 raft_consensus.cc:374] T 8a29996e3d594ebebae8b5fe7e02aa8c P b550bed5a64f466c8ba5a1761f3bef39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:40.610599 28321 raft_consensus.cc:729] T 8a29996e3d594ebebae8b5fe7e02aa8c P b550bed5a64f466c8ba5a1761f3bef39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b550bed5a64f466c8ba5a1761f3bef39, State: Initialized, Role: FOLLOWER
I20240502 01:55:40.610874 28321 consensus_queue.cc:260] T 8a29996e3d594ebebae8b5fe7e02aa8c P b550bed5a64f466c8ba5a1761f3bef39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } }
I20240502 01:55:40.611469 28321 ts_tablet_manager.cc:1398] T 8a29996e3d594ebebae8b5fe7e02aa8c P b550bed5a64f466c8ba5a1761f3bef39: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 01:55:40.611812 28321 tablet_bootstrap.cc:492] T 9a16d465aa2b426484868a2b212c5d00 P b550bed5a64f466c8ba5a1761f3bef39: Bootstrap starting.
I20240502 01:55:40.613708 28321 tablet_bootstrap.cc:654] T 9a16d465aa2b426484868a2b212c5d00 P b550bed5a64f466c8ba5a1761f3bef39: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:40.616047 28320 tablet_bootstrap.cc:492] T f1597c56a1a74b37bb5e18e8c4662f46 P d50f07af87d74f039fb2b43cf166fa43: No bootstrap required, opened a new log
I20240502 01:55:40.616248 28320 ts_tablet_manager.cc:1367] T f1597c56a1a74b37bb5e18e8c4662f46 P d50f07af87d74f039fb2b43cf166fa43: Time spent bootstrapping tablet: real 0.013s	user 0.004s	sys 0.000s
I20240502 01:55:40.616279 28321 tablet_bootstrap.cc:492] T 9a16d465aa2b426484868a2b212c5d00 P b550bed5a64f466c8ba5a1761f3bef39: No bootstrap required, opened a new log
I20240502 01:55:40.616446 28321 ts_tablet_manager.cc:1367] T 9a16d465aa2b426484868a2b212c5d00 P b550bed5a64f466c8ba5a1761f3bef39: Time spent bootstrapping tablet: real 0.005s	user 0.002s	sys 0.002s
I20240502 01:55:40.616935 28320 raft_consensus.cc:348] T f1597c56a1a74b37bb5e18e8c4662f46 P d50f07af87d74f039fb2b43cf166fa43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } }
I20240502 01:55:40.617146 28320 raft_consensus.cc:374] T f1597c56a1a74b37bb5e18e8c4662f46 P d50f07af87d74f039fb2b43cf166fa43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:40.617031 28321 raft_consensus.cc:348] T 9a16d465aa2b426484868a2b212c5d00 P b550bed5a64f466c8ba5a1761f3bef39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } }
I20240502 01:55:40.617223 28320 raft_consensus.cc:729] T f1597c56a1a74b37bb5e18e8c4662f46 P d50f07af87d74f039fb2b43cf166fa43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d50f07af87d74f039fb2b43cf166fa43, State: Initialized, Role: FOLLOWER
I20240502 01:55:40.617257 28321 raft_consensus.cc:374] T 9a16d465aa2b426484868a2b212c5d00 P b550bed5a64f466c8ba5a1761f3bef39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:40.617331 28321 raft_consensus.cc:729] T 9a16d465aa2b426484868a2b212c5d00 P b550bed5a64f466c8ba5a1761f3bef39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b550bed5a64f466c8ba5a1761f3bef39, State: Initialized, Role: FOLLOWER
I20240502 01:55:40.617496 28320 consensus_queue.cc:260] T f1597c56a1a74b37bb5e18e8c4662f46 P d50f07af87d74f039fb2b43cf166fa43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } }
I20240502 01:55:40.617601 28321 consensus_queue.cc:260] T 9a16d465aa2b426484868a2b212c5d00 P b550bed5a64f466c8ba5a1761f3bef39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } }
I20240502 01:55:40.618090 28320 ts_tablet_manager.cc:1398] T f1597c56a1a74b37bb5e18e8c4662f46 P d50f07af87d74f039fb2b43cf166fa43: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 01:55:40.618175 28321 ts_tablet_manager.cc:1398] T 9a16d465aa2b426484868a2b212c5d00 P b550bed5a64f466c8ba5a1761f3bef39: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 01:55:40.646128 28326 raft_consensus.cc:482] T f1597c56a1a74b37bb5e18e8c4662f46 P a64152fd9595478ca767a9f248ef3b85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:55:40.646427 28326 raft_consensus.cc:504] T f1597c56a1a74b37bb5e18e8c4662f46 P a64152fd9595478ca767a9f248ef3b85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } }
I20240502 01:55:40.648142 28326 leader_election.cc:288] T f1597c56a1a74b37bb5e18e8c4662f46 P a64152fd9595478ca767a9f248ef3b85 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d50f07af87d74f039fb2b43cf166fa43 (127.9.176.193:34203), b550bed5a64f466c8ba5a1761f3bef39 (127.9.176.195:40013)
I20240502 01:55:40.655928 28326 raft_consensus.cc:482] T 8a29996e3d594ebebae8b5fe7e02aa8c P a64152fd9595478ca767a9f248ef3b85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:55:40.656176 28326 raft_consensus.cc:504] T 8a29996e3d594ebebae8b5fe7e02aa8c P a64152fd9595478ca767a9f248ef3b85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } }
I20240502 01:55:40.656549 27983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1597c56a1a74b37bb5e18e8c4662f46" candidate_uuid: "a64152fd9595478ca767a9f248ef3b85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b550bed5a64f466c8ba5a1761f3bef39" is_pre_election: true
I20240502 01:55:40.657001 28326 leader_election.cc:288] T 8a29996e3d594ebebae8b5fe7e02aa8c P a64152fd9595478ca767a9f248ef3b85 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d50f07af87d74f039fb2b43cf166fa43 (127.9.176.193:34203), b550bed5a64f466c8ba5a1761f3bef39 (127.9.176.195:40013)
I20240502 01:55:40.656877 27716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1597c56a1a74b37bb5e18e8c4662f46" candidate_uuid: "a64152fd9595478ca767a9f248ef3b85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d50f07af87d74f039fb2b43cf166fa43" is_pre_election: true
I20240502 01:55:40.657056 27983 raft_consensus.cc:2454] T f1597c56a1a74b37bb5e18e8c4662f46 P b550bed5a64f466c8ba5a1761f3bef39 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64152fd9595478ca767a9f248ef3b85 in term 0.
I20240502 01:55:40.657409 27716 raft_consensus.cc:2454] T f1597c56a1a74b37bb5e18e8c4662f46 P d50f07af87d74f039fb2b43cf166fa43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64152fd9595478ca767a9f248ef3b85 in term 0.
I20240502 01:55:40.657784 27784 leader_election.cc:302] T f1597c56a1a74b37bb5e18e8c4662f46 P a64152fd9595478ca767a9f248ef3b85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64152fd9595478ca767a9f248ef3b85, b550bed5a64f466c8ba5a1761f3bef39; no voters: 
I20240502 01:55:40.657850 27983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a29996e3d594ebebae8b5fe7e02aa8c" candidate_uuid: "a64152fd9595478ca767a9f248ef3b85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b550bed5a64f466c8ba5a1761f3bef39" is_pre_election: true
I20240502 01:55:40.657742 27715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a29996e3d594ebebae8b5fe7e02aa8c" candidate_uuid: "a64152fd9595478ca767a9f248ef3b85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d50f07af87d74f039fb2b43cf166fa43" is_pre_election: true
I20240502 01:55:40.658059 27983 raft_consensus.cc:2454] T 8a29996e3d594ebebae8b5fe7e02aa8c P b550bed5a64f466c8ba5a1761f3bef39 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64152fd9595478ca767a9f248ef3b85 in term 0.
I20240502 01:55:40.658078 27715 raft_consensus.cc:2454] T 8a29996e3d594ebebae8b5fe7e02aa8c P d50f07af87d74f039fb2b43cf166fa43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64152fd9595478ca767a9f248ef3b85 in term 0.
I20240502 01:55:40.658268 28326 raft_consensus.cc:2781] T f1597c56a1a74b37bb5e18e8c4662f46 P a64152fd9595478ca767a9f248ef3b85 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:55:40.658396 28326 raft_consensus.cc:482] T f1597c56a1a74b37bb5e18e8c4662f46 P a64152fd9595478ca767a9f248ef3b85 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:55:40.658517 27783 leader_election.cc:302] T 8a29996e3d594ebebae8b5fe7e02aa8c P a64152fd9595478ca767a9f248ef3b85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64152fd9595478ca767a9f248ef3b85, d50f07af87d74f039fb2b43cf166fa43; no voters: 
I20240502 01:55:40.658541 28326 raft_consensus.cc:3037] T f1597c56a1a74b37bb5e18e8c4662f46 P a64152fd9595478ca767a9f248ef3b85 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:40.659371 28336 raft_consensus.cc:2781] T 8a29996e3d594ebebae8b5fe7e02aa8c P a64152fd9595478ca767a9f248ef3b85 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:55:40.659487 28336 raft_consensus.cc:482] T 8a29996e3d594ebebae8b5fe7e02aa8c P a64152fd9595478ca767a9f248ef3b85 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:55:40.659659 28336 raft_consensus.cc:3037] T 8a29996e3d594ebebae8b5fe7e02aa8c P a64152fd9595478ca767a9f248ef3b85 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:40.661131 28326 raft_consensus.cc:504] T f1597c56a1a74b37bb5e18e8c4662f46 P a64152fd9595478ca767a9f248ef3b85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } }
I20240502 01:55:40.661780 28326 leader_election.cc:288] T f1597c56a1a74b37bb5e18e8c4662f46 P a64152fd9595478ca767a9f248ef3b85 [CANDIDATE]: Term 1 election: Requested vote from peers d50f07af87d74f039fb2b43cf166fa43 (127.9.176.193:34203), b550bed5a64f466c8ba5a1761f3bef39 (127.9.176.195:40013)
I20240502 01:55:40.661871 28336 raft_consensus.cc:504] T 8a29996e3d594ebebae8b5fe7e02aa8c P a64152fd9595478ca767a9f248ef3b85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } }
I20240502 01:55:40.662357 27715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1597c56a1a74b37bb5e18e8c4662f46" candidate_uuid: "a64152fd9595478ca767a9f248ef3b85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d50f07af87d74f039fb2b43cf166fa43"
I20240502 01:55:40.662366 27983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1597c56a1a74b37bb5e18e8c4662f46" candidate_uuid: "a64152fd9595478ca767a9f248ef3b85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b550bed5a64f466c8ba5a1761f3bef39"
I20240502 01:55:40.662557 27983 raft_consensus.cc:3037] T f1597c56a1a74b37bb5e18e8c4662f46 P b550bed5a64f466c8ba5a1761f3bef39 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:40.662540 27715 raft_consensus.cc:3037] T f1597c56a1a74b37bb5e18e8c4662f46 P d50f07af87d74f039fb2b43cf166fa43 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:40.664741 27715 raft_consensus.cc:2454] T f1597c56a1a74b37bb5e18e8c4662f46 P d50f07af87d74f039fb2b43cf166fa43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64152fd9595478ca767a9f248ef3b85 in term 1.
I20240502 01:55:40.664758 27983 raft_consensus.cc:2454] T f1597c56a1a74b37bb5e18e8c4662f46 P b550bed5a64f466c8ba5a1761f3bef39 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64152fd9595478ca767a9f248ef3b85 in term 1.
I20240502 01:55:40.665228 27783 leader_election.cc:302] T f1597c56a1a74b37bb5e18e8c4662f46 P a64152fd9595478ca767a9f248ef3b85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64152fd9595478ca767a9f248ef3b85, d50f07af87d74f039fb2b43cf166fa43; no voters: 
I20240502 01:55:40.665472 28336 leader_election.cc:288] T 8a29996e3d594ebebae8b5fe7e02aa8c P a64152fd9595478ca767a9f248ef3b85 [CANDIDATE]: Term 1 election: Requested vote from peers d50f07af87d74f039fb2b43cf166fa43 (127.9.176.193:34203), b550bed5a64f466c8ba5a1761f3bef39 (127.9.176.195:40013)
I20240502 01:55:40.665936 28326 raft_consensus.cc:2781] T f1597c56a1a74b37bb5e18e8c4662f46 P a64152fd9595478ca767a9f248ef3b85 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:55:40.666105 27983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a29996e3d594ebebae8b5fe7e02aa8c" candidate_uuid: "a64152fd9595478ca767a9f248ef3b85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b550bed5a64f466c8ba5a1761f3bef39"
I20240502 01:55:40.666244 28326 raft_consensus.cc:686] T f1597c56a1a74b37bb5e18e8c4662f46 P a64152fd9595478ca767a9f248ef3b85 [term 1 LEADER]: Becoming Leader. State: Replica: a64152fd9595478ca767a9f248ef3b85, State: Running, Role: LEADER
I20240502 01:55:40.666365 27983 raft_consensus.cc:3037] T 8a29996e3d594ebebae8b5fe7e02aa8c P b550bed5a64f466c8ba5a1761f3bef39 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:40.666152 27715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a29996e3d594ebebae8b5fe7e02aa8c" candidate_uuid: "a64152fd9595478ca767a9f248ef3b85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d50f07af87d74f039fb2b43cf166fa43"
I20240502 01:55:40.666709 27715 raft_consensus.cc:3037] T 8a29996e3d594ebebae8b5fe7e02aa8c P d50f07af87d74f039fb2b43cf166fa43 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:40.666826 28326 consensus_queue.cc:237] T f1597c56a1a74b37bb5e18e8c4662f46 P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } }
I20240502 01:55:40.668586 27983 raft_consensus.cc:2454] T 8a29996e3d594ebebae8b5fe7e02aa8c P b550bed5a64f466c8ba5a1761f3bef39 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64152fd9595478ca767a9f248ef3b85 in term 1.
I20240502 01:55:40.669970 27784 leader_election.cc:302] T 8a29996e3d594ebebae8b5fe7e02aa8c P a64152fd9595478ca767a9f248ef3b85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64152fd9595478ca767a9f248ef3b85, b550bed5a64f466c8ba5a1761f3bef39; no voters: 
I20240502 01:55:40.670377 28336 raft_consensus.cc:2781] T 8a29996e3d594ebebae8b5fe7e02aa8c P a64152fd9595478ca767a9f248ef3b85 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:55:40.669553 27715 raft_consensus.cc:2454] T 8a29996e3d594ebebae8b5fe7e02aa8c P d50f07af87d74f039fb2b43cf166fa43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64152fd9595478ca767a9f248ef3b85 in term 1.
I20240502 01:55:40.670929 28336 raft_consensus.cc:686] T 8a29996e3d594ebebae8b5fe7e02aa8c P a64152fd9595478ca767a9f248ef3b85 [term 1 LEADER]: Becoming Leader. State: Replica: a64152fd9595478ca767a9f248ef3b85, State: Running, Role: LEADER
I20240502 01:55:40.671309 28336 consensus_queue.cc:237] T 8a29996e3d594ebebae8b5fe7e02aa8c P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } }
I20240502 01:55:40.674741 27573 catalog_manager.cc:5484] T f1597c56a1a74b37bb5e18e8c4662f46 P a64152fd9595478ca767a9f248ef3b85 reported cstate change: term changed from 0 to 1, leader changed from <none> to a64152fd9595478ca767a9f248ef3b85 (127.9.176.194). New cstate: current_term: 1 leader_uuid: "a64152fd9595478ca767a9f248ef3b85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } health_report { overall_health: UNKNOWN } } }
I20240502 01:55:40.680811 27573 catalog_manager.cc:5484] T 8a29996e3d594ebebae8b5fe7e02aa8c P a64152fd9595478ca767a9f248ef3b85 reported cstate change: term changed from 0 to 1, leader changed from <none> to a64152fd9595478ca767a9f248ef3b85 (127.9.176.194). New cstate: current_term: 1 leader_uuid: "a64152fd9595478ca767a9f248ef3b85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } health_report { overall_health: UNKNOWN } } }
W20240502 01:55:40.824177 27763 tablet.cc:2369] T 8a29996e3d594ebebae8b5fe7e02aa8c P d50f07af87d74f039fb2b43cf166fa43: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:55:40.847250 28325 raft_consensus.cc:482] T 9a16d465aa2b426484868a2b212c5d00 P d50f07af87d74f039fb2b43cf166fa43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:55:40.847491 28325 raft_consensus.cc:504] T 9a16d465aa2b426484868a2b212c5d00 P d50f07af87d74f039fb2b43cf166fa43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } }
I20240502 01:55:40.849190 28325 leader_election.cc:288] T 9a16d465aa2b426484868a2b212c5d00 P d50f07af87d74f039fb2b43cf166fa43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64152fd9595478ca767a9f248ef3b85 (127.9.176.194:38689), b550bed5a64f466c8ba5a1761f3bef39 (127.9.176.195:40013)
I20240502 01:55:40.857632 28328 raft_consensus.cc:482] T 9a16d465aa2b426484868a2b212c5d00 P b550bed5a64f466c8ba5a1761f3bef39 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:55:40.857894 28328 raft_consensus.cc:504] T 9a16d465aa2b426484868a2b212c5d00 P b550bed5a64f466c8ba5a1761f3bef39 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } }
I20240502 01:55:40.858872 27983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a16d465aa2b426484868a2b212c5d00" candidate_uuid: "d50f07af87d74f039fb2b43cf166fa43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b550bed5a64f466c8ba5a1761f3bef39" is_pre_election: true
I20240502 01:55:40.859119 27983 raft_consensus.cc:2454] T 9a16d465aa2b426484868a2b212c5d00 P b550bed5a64f466c8ba5a1761f3bef39 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d50f07af87d74f039fb2b43cf166fa43 in term 0.
I20240502 01:55:40.859738 27651 leader_election.cc:302] T 9a16d465aa2b426484868a2b212c5d00 P d50f07af87d74f039fb2b43cf166fa43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b550bed5a64f466c8ba5a1761f3bef39, d50f07af87d74f039fb2b43cf166fa43; no voters: 
I20240502 01:55:40.860210 28328 leader_election.cc:288] T 9a16d465aa2b426484868a2b212c5d00 P b550bed5a64f466c8ba5a1761f3bef39 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d50f07af87d74f039fb2b43cf166fa43 (127.9.176.193:34203), a64152fd9595478ca767a9f248ef3b85 (127.9.176.194:38689)
I20240502 01:55:40.860283 27849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a16d465aa2b426484868a2b212c5d00" candidate_uuid: "d50f07af87d74f039fb2b43cf166fa43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64152fd9595478ca767a9f248ef3b85" is_pre_election: true
I20240502 01:55:40.860736 27849 raft_consensus.cc:2454] T 9a16d465aa2b426484868a2b212c5d00 P a64152fd9595478ca767a9f248ef3b85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d50f07af87d74f039fb2b43cf166fa43 in term 0.
I20240502 01:55:40.861155 28325 raft_consensus.cc:2781] T 9a16d465aa2b426484868a2b212c5d00 P d50f07af87d74f039fb2b43cf166fa43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:55:40.861296 28325 raft_consensus.cc:482] T 9a16d465aa2b426484868a2b212c5d00 P d50f07af87d74f039fb2b43cf166fa43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:55:40.861400 28325 raft_consensus.cc:3037] T 9a16d465aa2b426484868a2b212c5d00 P d50f07af87d74f039fb2b43cf166fa43 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:40.864547 28325 raft_consensus.cc:504] T 9a16d465aa2b426484868a2b212c5d00 P d50f07af87d74f039fb2b43cf166fa43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } }
I20240502 01:55:40.865444 28325 leader_election.cc:288] T 9a16d465aa2b426484868a2b212c5d00 P d50f07af87d74f039fb2b43cf166fa43 [CANDIDATE]: Term 1 election: Requested vote from peers a64152fd9595478ca767a9f248ef3b85 (127.9.176.194:38689), b550bed5a64f466c8ba5a1761f3bef39 (127.9.176.195:40013)
I20240502 01:55:40.866039 27849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a16d465aa2b426484868a2b212c5d00" candidate_uuid: "d50f07af87d74f039fb2b43cf166fa43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64152fd9595478ca767a9f248ef3b85"
I20240502 01:55:40.866299 27849 raft_consensus.cc:3037] T 9a16d465aa2b426484868a2b212c5d00 P a64152fd9595478ca767a9f248ef3b85 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:40.866395 27983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a16d465aa2b426484868a2b212c5d00" candidate_uuid: "d50f07af87d74f039fb2b43cf166fa43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b550bed5a64f466c8ba5a1761f3bef39"
I20240502 01:55:40.866659 27983 raft_consensus.cc:3037] T 9a16d465aa2b426484868a2b212c5d00 P b550bed5a64f466c8ba5a1761f3bef39 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:40.869995 27849 raft_consensus.cc:2454] T 9a16d465aa2b426484868a2b212c5d00 P a64152fd9595478ca767a9f248ef3b85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d50f07af87d74f039fb2b43cf166fa43 in term 1.
I20240502 01:55:40.870443 27983 raft_consensus.cc:2454] T 9a16d465aa2b426484868a2b212c5d00 P b550bed5a64f466c8ba5a1761f3bef39 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d50f07af87d74f039fb2b43cf166fa43 in term 1.
I20240502 01:55:40.870904 27849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a16d465aa2b426484868a2b212c5d00" candidate_uuid: "b550bed5a64f466c8ba5a1761f3bef39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64152fd9595478ca767a9f248ef3b85" is_pre_election: true
I20240502 01:55:40.871006 27715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a16d465aa2b426484868a2b212c5d00" candidate_uuid: "b550bed5a64f466c8ba5a1761f3bef39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d50f07af87d74f039fb2b43cf166fa43" is_pre_election: true
I20240502 01:55:40.871261 27849 raft_consensus.cc:2379] T 9a16d465aa2b426484868a2b212c5d00 P a64152fd9595478ca767a9f248ef3b85 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b550bed5a64f466c8ba5a1761f3bef39 in current term 1: Already voted for candidate d50f07af87d74f039fb2b43cf166fa43 in this term.
I20240502 01:55:40.871326 27715 raft_consensus.cc:2379] T 9a16d465aa2b426484868a2b212c5d00 P d50f07af87d74f039fb2b43cf166fa43 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b550bed5a64f466c8ba5a1761f3bef39 in current term 1: Already voted for candidate d50f07af87d74f039fb2b43cf166fa43 in this term.
I20240502 01:55:40.870738 27651 leader_election.cc:302] T 9a16d465aa2b426484868a2b212c5d00 P d50f07af87d74f039fb2b43cf166fa43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64152fd9595478ca767a9f248ef3b85, d50f07af87d74f039fb2b43cf166fa43; no voters: 
I20240502 01:55:40.872014 28325 raft_consensus.cc:2781] T 9a16d465aa2b426484868a2b212c5d00 P d50f07af87d74f039fb2b43cf166fa43 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:55:40.872382 27917 leader_election.cc:302] T 9a16d465aa2b426484868a2b212c5d00 P b550bed5a64f466c8ba5a1761f3bef39 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b550bed5a64f466c8ba5a1761f3bef39; no voters: a64152fd9595478ca767a9f248ef3b85, d50f07af87d74f039fb2b43cf166fa43
I20240502 01:55:40.872629 28325 raft_consensus.cc:686] T 9a16d465aa2b426484868a2b212c5d00 P d50f07af87d74f039fb2b43cf166fa43 [term 1 LEADER]: Becoming Leader. State: Replica: d50f07af87d74f039fb2b43cf166fa43, State: Running, Role: LEADER
I20240502 01:55:40.872947 28328 raft_consensus.cc:2726] T 9a16d465aa2b426484868a2b212c5d00 P b550bed5a64f466c8ba5a1761f3bef39 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:55:40.873308 28325 consensus_queue.cc:237] T 9a16d465aa2b426484868a2b212c5d00 P d50f07af87d74f039fb2b43cf166fa43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } }
I20240502 01:55:40.880930 27573 catalog_manager.cc:5484] T 9a16d465aa2b426484868a2b212c5d00 P d50f07af87d74f039fb2b43cf166fa43 reported cstate change: term changed from 0 to 1, leader changed from <none> to d50f07af87d74f039fb2b43cf166fa43 (127.9.176.193). New cstate: current_term: 1 leader_uuid: "d50f07af87d74f039fb2b43cf166fa43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } health_report { overall_health: UNKNOWN } } }
I20240502 01:55:41.135529 28336 consensus_queue.cc:1035] T 8a29996e3d594ebebae8b5fe7e02aa8c P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:41.144733 28336 consensus_queue.cc:1035] T 8a29996e3d594ebebae8b5fe7e02aa8c P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:41.211649 28336 consensus_queue.cc:1035] T f1597c56a1a74b37bb5e18e8c4662f46 P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:41.218657 28326 consensus_queue.cc:1035] T f1597c56a1a74b37bb5e18e8c4662f46 P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:41.327858 28325 consensus_queue.cc:1035] T 9a16d465aa2b426484868a2b212c5d00 P d50f07af87d74f039fb2b43cf166fa43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:41.336277 28325 consensus_queue.cc:1035] T 9a16d465aa2b426484868a2b212c5d00 P d50f07af87d74f039fb2b43cf166fa43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:55:41.361308 28346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:55:41.361761 28346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:55:41.369711 28346 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'

INSERT report
    rows total: 2000
    time total: 274.938 ms
  time per row: 0.137469 ms
I20240502 01:55:41.804374 27574 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41110:
name: "rebalance_test_table_1"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
W20240502 01:55:41.805235 27574 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 01:55:41.815102 27827 tablet_service.cc:1467] Processing CreateTablet for tablet 63136d6964084523bb950cc6568a273b (DEFAULT_TABLE table=rebalance_test_table_1 [id=fdfd057dc2884c01bb7ee4631f5e3566]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:41.815141 27695 tablet_service.cc:1467] Processing CreateTablet for tablet 63136d6964084523bb950cc6568a273b (DEFAULT_TABLE table=rebalance_test_table_1 [id=fdfd057dc2884c01bb7ee4631f5e3566]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:41.815687 27695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63136d6964084523bb950cc6568a273b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:41.815694 27827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63136d6964084523bb950cc6568a273b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:41.815495 27696 tablet_service.cc:1467] Processing CreateTablet for tablet 5e8b27dca9af473188c50269820343f1 (DEFAULT_TABLE table=rebalance_test_table_1 [id=fdfd057dc2884c01bb7ee4631f5e3566]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:41.816184 27696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e8b27dca9af473188c50269820343f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:41.817337 27960 tablet_service.cc:1467] Processing CreateTablet for tablet 5e8b27dca9af473188c50269820343f1 (DEFAULT_TABLE table=rebalance_test_table_1 [id=fdfd057dc2884c01bb7ee4631f5e3566]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:41.817796 27960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e8b27dca9af473188c50269820343f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:41.817978 27961 tablet_service.cc:1467] Processing CreateTablet for tablet f22c0dfa05664a4a9f10aa81424a396e (DEFAULT_TABLE table=rebalance_test_table_1 [id=fdfd057dc2884c01bb7ee4631f5e3566]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:41.818428 27961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f22c0dfa05664a4a9f10aa81424a396e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:41.820053 27962 tablet_service.cc:1467] Processing CreateTablet for tablet 63136d6964084523bb950cc6568a273b (DEFAULT_TABLE table=rebalance_test_table_1 [id=fdfd057dc2884c01bb7ee4631f5e3566]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:41.820456 27962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63136d6964084523bb950cc6568a273b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:41.821256 27829 tablet_service.cc:1467] Processing CreateTablet for tablet 5e8b27dca9af473188c50269820343f1 (DEFAULT_TABLE table=rebalance_test_table_1 [id=fdfd057dc2884c01bb7ee4631f5e3566]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:41.821669 27829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e8b27dca9af473188c50269820343f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:41.822305 27694 tablet_service.cc:1467] Processing CreateTablet for tablet f22c0dfa05664a4a9f10aa81424a396e (DEFAULT_TABLE table=rebalance_test_table_1 [id=fdfd057dc2884c01bb7ee4631f5e3566]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:41.822705 27694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f22c0dfa05664a4a9f10aa81424a396e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:41.828461 27828 tablet_service.cc:1467] Processing CreateTablet for tablet f22c0dfa05664a4a9f10aa81424a396e (DEFAULT_TABLE table=rebalance_test_table_1 [id=fdfd057dc2884c01bb7ee4631f5e3566]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:41.829144 27828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f22c0dfa05664a4a9f10aa81424a396e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:41.840590 28411 tablet_bootstrap.cc:492] T f22c0dfa05664a4a9f10aa81424a396e P b550bed5a64f466c8ba5a1761f3bef39: Bootstrap starting.
I20240502 01:55:41.842563 28411 tablet_bootstrap.cc:654] T f22c0dfa05664a4a9f10aa81424a396e P b550bed5a64f466c8ba5a1761f3bef39: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:41.843106 28412 tablet_bootstrap.cc:492] T 63136d6964084523bb950cc6568a273b P d50f07af87d74f039fb2b43cf166fa43: Bootstrap starting.
I20240502 01:55:41.848614 28410 tablet_bootstrap.cc:492] T 63136d6964084523bb950cc6568a273b P a64152fd9595478ca767a9f248ef3b85: Bootstrap starting.
I20240502 01:55:41.850183 28411 tablet_bootstrap.cc:492] T f22c0dfa05664a4a9f10aa81424a396e P b550bed5a64f466c8ba5a1761f3bef39: No bootstrap required, opened a new log
I20240502 01:55:41.850451 28411 ts_tablet_manager.cc:1367] T f22c0dfa05664a4a9f10aa81424a396e P b550bed5a64f466c8ba5a1761f3bef39: Time spent bootstrapping tablet: real 0.010s	user 0.008s	sys 0.000s
I20240502 01:55:41.851044 28410 tablet_bootstrap.cc:654] T 63136d6964084523bb950cc6568a273b P a64152fd9595478ca767a9f248ef3b85: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:41.851413 28411 raft_consensus.cc:348] T f22c0dfa05664a4a9f10aa81424a396e P b550bed5a64f466c8ba5a1761f3bef39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } }
I20240502 01:55:41.851738 28411 raft_consensus.cc:374] T f22c0dfa05664a4a9f10aa81424a396e P b550bed5a64f466c8ba5a1761f3bef39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:41.851897 28411 raft_consensus.cc:729] T f22c0dfa05664a4a9f10aa81424a396e P b550bed5a64f466c8ba5a1761f3bef39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b550bed5a64f466c8ba5a1761f3bef39, State: Initialized, Role: FOLLOWER
I20240502 01:55:41.852208 28411 consensus_queue.cc:260] T f22c0dfa05664a4a9f10aa81424a396e P b550bed5a64f466c8ba5a1761f3bef39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } }
I20240502 01:55:41.853408 28411 ts_tablet_manager.cc:1398] T f22c0dfa05664a4a9f10aa81424a396e P b550bed5a64f466c8ba5a1761f3bef39: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 01:55:41.853910 28411 tablet_bootstrap.cc:492] T 63136d6964084523bb950cc6568a273b P b550bed5a64f466c8ba5a1761f3bef39: Bootstrap starting.
I20240502 01:55:41.854542 28410 tablet_bootstrap.cc:492] T 63136d6964084523bb950cc6568a273b P a64152fd9595478ca767a9f248ef3b85: No bootstrap required, opened a new log
I20240502 01:55:41.854658 28412 tablet_bootstrap.cc:654] T 63136d6964084523bb950cc6568a273b P d50f07af87d74f039fb2b43cf166fa43: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:41.854717 28410 ts_tablet_manager.cc:1367] T 63136d6964084523bb950cc6568a273b P a64152fd9595478ca767a9f248ef3b85: Time spent bootstrapping tablet: real 0.006s	user 0.005s	sys 0.000s
I20240502 01:55:41.855677 28410 raft_consensus.cc:348] T 63136d6964084523bb950cc6568a273b P a64152fd9595478ca767a9f248ef3b85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } }
I20240502 01:55:41.855953 28410 raft_consensus.cc:374] T 63136d6964084523bb950cc6568a273b P a64152fd9595478ca767a9f248ef3b85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:41.856051 28410 raft_consensus.cc:729] T 63136d6964084523bb950cc6568a273b P a64152fd9595478ca767a9f248ef3b85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64152fd9595478ca767a9f248ef3b85, State: Initialized, Role: FOLLOWER
I20240502 01:55:41.856328 28410 consensus_queue.cc:260] T 63136d6964084523bb950cc6568a273b P a64152fd9595478ca767a9f248ef3b85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } }
I20240502 01:55:41.856719 28411 tablet_bootstrap.cc:654] T 63136d6964084523bb950cc6568a273b P b550bed5a64f466c8ba5a1761f3bef39: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:41.857115 28410 ts_tablet_manager.cc:1398] T 63136d6964084523bb950cc6568a273b P a64152fd9595478ca767a9f248ef3b85: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240502 01:55:41.857477 28410 tablet_bootstrap.cc:492] T f22c0dfa05664a4a9f10aa81424a396e P a64152fd9595478ca767a9f248ef3b85: Bootstrap starting.
I20240502 01:55:41.860123 28410 tablet_bootstrap.cc:654] T f22c0dfa05664a4a9f10aa81424a396e P a64152fd9595478ca767a9f248ef3b85: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:41.865221 28410 tablet_bootstrap.cc:492] T f22c0dfa05664a4a9f10aa81424a396e P a64152fd9595478ca767a9f248ef3b85: No bootstrap required, opened a new log
I20240502 01:55:41.865679 28410 ts_tablet_manager.cc:1367] T f22c0dfa05664a4a9f10aa81424a396e P a64152fd9595478ca767a9f248ef3b85: Time spent bootstrapping tablet: real 0.008s	user 0.003s	sys 0.004s
I20240502 01:55:41.867053 28410 raft_consensus.cc:348] T f22c0dfa05664a4a9f10aa81424a396e P a64152fd9595478ca767a9f248ef3b85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } }
I20240502 01:55:41.867605 28410 raft_consensus.cc:374] T f22c0dfa05664a4a9f10aa81424a396e P a64152fd9595478ca767a9f248ef3b85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:41.867801 28410 raft_consensus.cc:729] T f22c0dfa05664a4a9f10aa81424a396e P a64152fd9595478ca767a9f248ef3b85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64152fd9595478ca767a9f248ef3b85, State: Initialized, Role: FOLLOWER
I20240502 01:55:41.868275 28410 consensus_queue.cc:260] T f22c0dfa05664a4a9f10aa81424a396e P a64152fd9595478ca767a9f248ef3b85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } }
I20240502 01:55:41.869150 28410 ts_tablet_manager.cc:1398] T f22c0dfa05664a4a9f10aa81424a396e P a64152fd9595478ca767a9f248ef3b85: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.002s
I20240502 01:55:41.869720 28410 tablet_bootstrap.cc:492] T 5e8b27dca9af473188c50269820343f1 P a64152fd9595478ca767a9f248ef3b85: Bootstrap starting.
I20240502 01:55:41.872504 28410 tablet_bootstrap.cc:654] T 5e8b27dca9af473188c50269820343f1 P a64152fd9595478ca767a9f248ef3b85: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:41.873214 28412 tablet_bootstrap.cc:492] T 63136d6964084523bb950cc6568a273b P d50f07af87d74f039fb2b43cf166fa43: No bootstrap required, opened a new log
I20240502 01:55:41.873497 28412 ts_tablet_manager.cc:1367] T 63136d6964084523bb950cc6568a273b P d50f07af87d74f039fb2b43cf166fa43: Time spent bootstrapping tablet: real 0.031s	user 0.006s	sys 0.000s
I20240502 01:55:41.875751 28412 raft_consensus.cc:348] T 63136d6964084523bb950cc6568a273b P d50f07af87d74f039fb2b43cf166fa43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } }
I20240502 01:55:41.876106 28412 raft_consensus.cc:374] T 63136d6964084523bb950cc6568a273b P d50f07af87d74f039fb2b43cf166fa43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:41.876283 28412 raft_consensus.cc:729] T 63136d6964084523bb950cc6568a273b P d50f07af87d74f039fb2b43cf166fa43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d50f07af87d74f039fb2b43cf166fa43, State: Initialized, Role: FOLLOWER
I20240502 01:55:41.876691 28411 tablet_bootstrap.cc:492] T 63136d6964084523bb950cc6568a273b P b550bed5a64f466c8ba5a1761f3bef39: No bootstrap required, opened a new log
I20240502 01:55:41.876683 28412 consensus_queue.cc:260] T 63136d6964084523bb950cc6568a273b P d50f07af87d74f039fb2b43cf166fa43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } }
I20240502 01:55:41.876965 28411 ts_tablet_manager.cc:1367] T 63136d6964084523bb950cc6568a273b P b550bed5a64f466c8ba5a1761f3bef39: Time spent bootstrapping tablet: real 0.023s	user 0.005s	sys 0.016s
I20240502 01:55:41.877611 28412 ts_tablet_manager.cc:1398] T 63136d6964084523bb950cc6568a273b P d50f07af87d74f039fb2b43cf166fa43: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.000s
I20240502 01:55:41.878049 28410 tablet_bootstrap.cc:492] T 5e8b27dca9af473188c50269820343f1 P a64152fd9595478ca767a9f248ef3b85: No bootstrap required, opened a new log
I20240502 01:55:41.877923 28411 raft_consensus.cc:348] T 63136d6964084523bb950cc6568a273b P b550bed5a64f466c8ba5a1761f3bef39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } }
I20240502 01:55:41.878167 28411 raft_consensus.cc:374] T 63136d6964084523bb950cc6568a273b P b550bed5a64f466c8ba5a1761f3bef39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:41.878265 28410 ts_tablet_manager.cc:1367] T 5e8b27dca9af473188c50269820343f1 P a64152fd9595478ca767a9f248ef3b85: Time spent bootstrapping tablet: real 0.009s	user 0.004s	sys 0.001s
I20240502 01:55:41.878291 28411 raft_consensus.cc:729] T 63136d6964084523bb950cc6568a273b P b550bed5a64f466c8ba5a1761f3bef39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b550bed5a64f466c8ba5a1761f3bef39, State: Initialized, Role: FOLLOWER
I20240502 01:55:41.878625 28411 consensus_queue.cc:260] T 63136d6964084523bb950cc6568a273b P b550bed5a64f466c8ba5a1761f3bef39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } }
I20240502 01:55:41.879137 28410 raft_consensus.cc:348] T 5e8b27dca9af473188c50269820343f1 P a64152fd9595478ca767a9f248ef3b85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } }
I20240502 01:55:41.879387 28410 raft_consensus.cc:374] T 5e8b27dca9af473188c50269820343f1 P a64152fd9595478ca767a9f248ef3b85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:41.879498 28410 raft_consensus.cc:729] T 5e8b27dca9af473188c50269820343f1 P a64152fd9595478ca767a9f248ef3b85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64152fd9595478ca767a9f248ef3b85, State: Initialized, Role: FOLLOWER
I20240502 01:55:41.879827 28410 consensus_queue.cc:260] T 5e8b27dca9af473188c50269820343f1 P a64152fd9595478ca767a9f248ef3b85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } }
I20240502 01:55:41.880414 28412 tablet_bootstrap.cc:492] T f22c0dfa05664a4a9f10aa81424a396e P d50f07af87d74f039fb2b43cf166fa43: Bootstrap starting.
I20240502 01:55:41.881983 28411 ts_tablet_manager.cc:1398] T 63136d6964084523bb950cc6568a273b P b550bed5a64f466c8ba5a1761f3bef39: Time spent starting tablet: real 0.005s	user 0.001s	sys 0.001s
I20240502 01:55:41.883350 28412 tablet_bootstrap.cc:654] T f22c0dfa05664a4a9f10aa81424a396e P d50f07af87d74f039fb2b43cf166fa43: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:41.884912 28411 tablet_bootstrap.cc:492] T 5e8b27dca9af473188c50269820343f1 P b550bed5a64f466c8ba5a1761f3bef39: Bootstrap starting.
I20240502 01:55:41.884963 28410 ts_tablet_manager.cc:1398] T 5e8b27dca9af473188c50269820343f1 P a64152fd9595478ca767a9f248ef3b85: Time spent starting tablet: real 0.007s	user 0.002s	sys 0.000s
I20240502 01:55:41.889743 28411 tablet_bootstrap.cc:654] T 5e8b27dca9af473188c50269820343f1 P b550bed5a64f466c8ba5a1761f3bef39: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:41.891559 28412 tablet_bootstrap.cc:492] T f22c0dfa05664a4a9f10aa81424a396e P d50f07af87d74f039fb2b43cf166fa43: No bootstrap required, opened a new log
I20240502 01:55:41.891815 28412 ts_tablet_manager.cc:1367] T f22c0dfa05664a4a9f10aa81424a396e P d50f07af87d74f039fb2b43cf166fa43: Time spent bootstrapping tablet: real 0.011s	user 0.004s	sys 0.006s
I20240502 01:55:41.892047 28411 tablet_bootstrap.cc:492] T 5e8b27dca9af473188c50269820343f1 P b550bed5a64f466c8ba5a1761f3bef39: No bootstrap required, opened a new log
I20240502 01:55:41.892210 28411 ts_tablet_manager.cc:1367] T 5e8b27dca9af473188c50269820343f1 P b550bed5a64f466c8ba5a1761f3bef39: Time spent bootstrapping tablet: real 0.007s	user 0.002s	sys 0.002s
I20240502 01:55:41.892678 28412 raft_consensus.cc:348] T f22c0dfa05664a4a9f10aa81424a396e P d50f07af87d74f039fb2b43cf166fa43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } }
I20240502 01:55:41.892886 28412 raft_consensus.cc:374] T f22c0dfa05664a4a9f10aa81424a396e P d50f07af87d74f039fb2b43cf166fa43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:41.892968 28412 raft_consensus.cc:729] T f22c0dfa05664a4a9f10aa81424a396e P d50f07af87d74f039fb2b43cf166fa43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d50f07af87d74f039fb2b43cf166fa43, State: Initialized, Role: FOLLOWER
I20240502 01:55:41.892939 28411 raft_consensus.cc:348] T 5e8b27dca9af473188c50269820343f1 P b550bed5a64f466c8ba5a1761f3bef39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } }
I20240502 01:55:41.893170 28411 raft_consensus.cc:374] T 5e8b27dca9af473188c50269820343f1 P b550bed5a64f466c8ba5a1761f3bef39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:41.893280 28411 raft_consensus.cc:729] T 5e8b27dca9af473188c50269820343f1 P b550bed5a64f466c8ba5a1761f3bef39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b550bed5a64f466c8ba5a1761f3bef39, State: Initialized, Role: FOLLOWER
I20240502 01:55:41.893318 28412 consensus_queue.cc:260] T f22c0dfa05664a4a9f10aa81424a396e P d50f07af87d74f039fb2b43cf166fa43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } }
I20240502 01:55:41.893546 28411 consensus_queue.cc:260] T 5e8b27dca9af473188c50269820343f1 P b550bed5a64f466c8ba5a1761f3bef39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } }
I20240502 01:55:41.893927 28412 ts_tablet_manager.cc:1398] T f22c0dfa05664a4a9f10aa81424a396e P d50f07af87d74f039fb2b43cf166fa43: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 01:55:41.894310 28412 tablet_bootstrap.cc:492] T 5e8b27dca9af473188c50269820343f1 P d50f07af87d74f039fb2b43cf166fa43: Bootstrap starting.
I20240502 01:55:41.894765 28411 ts_tablet_manager.cc:1398] T 5e8b27dca9af473188c50269820343f1 P b550bed5a64f466c8ba5a1761f3bef39: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 01:55:41.896760 28412 tablet_bootstrap.cc:654] T 5e8b27dca9af473188c50269820343f1 P d50f07af87d74f039fb2b43cf166fa43: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:41.898943 28412 tablet_bootstrap.cc:492] T 5e8b27dca9af473188c50269820343f1 P d50f07af87d74f039fb2b43cf166fa43: No bootstrap required, opened a new log
I20240502 01:55:41.899175 28412 ts_tablet_manager.cc:1367] T 5e8b27dca9af473188c50269820343f1 P d50f07af87d74f039fb2b43cf166fa43: Time spent bootstrapping tablet: real 0.005s	user 0.004s	sys 0.000s
I20240502 01:55:41.900072 28412 raft_consensus.cc:348] T 5e8b27dca9af473188c50269820343f1 P d50f07af87d74f039fb2b43cf166fa43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } }
I20240502 01:55:41.900313 28412 raft_consensus.cc:374] T 5e8b27dca9af473188c50269820343f1 P d50f07af87d74f039fb2b43cf166fa43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:41.900439 28412 raft_consensus.cc:729] T 5e8b27dca9af473188c50269820343f1 P d50f07af87d74f039fb2b43cf166fa43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d50f07af87d74f039fb2b43cf166fa43, State: Initialized, Role: FOLLOWER
I20240502 01:55:41.900717 28412 consensus_queue.cc:260] T 5e8b27dca9af473188c50269820343f1 P d50f07af87d74f039fb2b43cf166fa43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } }
I20240502 01:55:41.901450 28412 ts_tablet_manager.cc:1398] T 5e8b27dca9af473188c50269820343f1 P d50f07af87d74f039fb2b43cf166fa43: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240502 01:55:42.083381 28326 raft_consensus.cc:482] T 5e8b27dca9af473188c50269820343f1 P a64152fd9595478ca767a9f248ef3b85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:55:42.083675 28326 raft_consensus.cc:504] T 5e8b27dca9af473188c50269820343f1 P a64152fd9595478ca767a9f248ef3b85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } }
I20240502 01:55:42.084439 28326 leader_election.cc:288] T 5e8b27dca9af473188c50269820343f1 P a64152fd9595478ca767a9f248ef3b85 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d50f07af87d74f039fb2b43cf166fa43 (127.9.176.193:34203), b550bed5a64f466c8ba5a1761f3bef39 (127.9.176.195:40013)
I20240502 01:55:42.084452 28338 raft_consensus.cc:482] T 63136d6964084523bb950cc6568a273b P a64152fd9595478ca767a9f248ef3b85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:55:42.084681 28338 raft_consensus.cc:504] T 63136d6964084523bb950cc6568a273b P a64152fd9595478ca767a9f248ef3b85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } }
I20240502 01:55:42.085144 27716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e8b27dca9af473188c50269820343f1" candidate_uuid: "a64152fd9595478ca767a9f248ef3b85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d50f07af87d74f039fb2b43cf166fa43" is_pre_election: true
I20240502 01:55:42.085175 27982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e8b27dca9af473188c50269820343f1" candidate_uuid: "a64152fd9595478ca767a9f248ef3b85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b550bed5a64f466c8ba5a1761f3bef39" is_pre_election: true
I20240502 01:55:42.085407 28338 leader_election.cc:288] T 63136d6964084523bb950cc6568a273b P a64152fd9595478ca767a9f248ef3b85 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d50f07af87d74f039fb2b43cf166fa43 (127.9.176.193:34203), b550bed5a64f466c8ba5a1761f3bef39 (127.9.176.195:40013)
I20240502 01:55:42.085407 27716 raft_consensus.cc:2454] T 5e8b27dca9af473188c50269820343f1 P d50f07af87d74f039fb2b43cf166fa43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64152fd9595478ca767a9f248ef3b85 in term 0.
I20240502 01:55:42.085451 27982 raft_consensus.cc:2454] T 5e8b27dca9af473188c50269820343f1 P b550bed5a64f466c8ba5a1761f3bef39 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64152fd9595478ca767a9f248ef3b85 in term 0.
I20240502 01:55:42.085897 27784 leader_election.cc:302] T 5e8b27dca9af473188c50269820343f1 P a64152fd9595478ca767a9f248ef3b85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64152fd9595478ca767a9f248ef3b85, b550bed5a64f466c8ba5a1761f3bef39; no voters: 
I20240502 01:55:42.086041 27716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63136d6964084523bb950cc6568a273b" candidate_uuid: "a64152fd9595478ca767a9f248ef3b85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d50f07af87d74f039fb2b43cf166fa43" is_pre_election: true
I20240502 01:55:42.086179 27982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63136d6964084523bb950cc6568a273b" candidate_uuid: "a64152fd9595478ca767a9f248ef3b85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b550bed5a64f466c8ba5a1761f3bef39" is_pre_election: true
I20240502 01:55:42.086261 28338 raft_consensus.cc:2781] T 5e8b27dca9af473188c50269820343f1 P a64152fd9595478ca767a9f248ef3b85 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:55:42.086272 27716 raft_consensus.cc:2454] T 63136d6964084523bb950cc6568a273b P d50f07af87d74f039fb2b43cf166fa43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64152fd9595478ca767a9f248ef3b85 in term 0.
I20240502 01:55:42.086371 27982 raft_consensus.cc:2454] T 63136d6964084523bb950cc6568a273b P b550bed5a64f466c8ba5a1761f3bef39 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64152fd9595478ca767a9f248ef3b85 in term 0.
I20240502 01:55:42.086380 28338 raft_consensus.cc:482] T 5e8b27dca9af473188c50269820343f1 P a64152fd9595478ca767a9f248ef3b85 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:55:42.086472 28338 raft_consensus.cc:3037] T 5e8b27dca9af473188c50269820343f1 P a64152fd9595478ca767a9f248ef3b85 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:42.086704 27783 leader_election.cc:302] T 63136d6964084523bb950cc6568a273b P a64152fd9595478ca767a9f248ef3b85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64152fd9595478ca767a9f248ef3b85, d50f07af87d74f039fb2b43cf166fa43; no voters: 
I20240502 01:55:42.086982 28326 raft_consensus.cc:2781] T 63136d6964084523bb950cc6568a273b P a64152fd9595478ca767a9f248ef3b85 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:55:42.087097 28326 raft_consensus.cc:482] T 63136d6964084523bb950cc6568a273b P a64152fd9595478ca767a9f248ef3b85 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:55:42.087224 28326 raft_consensus.cc:3037] T 63136d6964084523bb950cc6568a273b P a64152fd9595478ca767a9f248ef3b85 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:42.088934 28338 raft_consensus.cc:504] T 5e8b27dca9af473188c50269820343f1 P a64152fd9595478ca767a9f248ef3b85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } }
I20240502 01:55:42.089365 28326 raft_consensus.cc:504] T 63136d6964084523bb950cc6568a273b P a64152fd9595478ca767a9f248ef3b85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } }
I20240502 01:55:42.089622 28338 leader_election.cc:288] T 5e8b27dca9af473188c50269820343f1 P a64152fd9595478ca767a9f248ef3b85 [CANDIDATE]: Term 1 election: Requested vote from peers d50f07af87d74f039fb2b43cf166fa43 (127.9.176.193:34203), b550bed5a64f466c8ba5a1761f3bef39 (127.9.176.195:40013)
I20240502 01:55:42.090220 28326 leader_election.cc:288] T 63136d6964084523bb950cc6568a273b P a64152fd9595478ca767a9f248ef3b85 [CANDIDATE]: Term 1 election: Requested vote from peers d50f07af87d74f039fb2b43cf166fa43 (127.9.176.193:34203), b550bed5a64f466c8ba5a1761f3bef39 (127.9.176.195:40013)
I20240502 01:55:42.090294 28387 raft_consensus.cc:482] T f22c0dfa05664a4a9f10aa81424a396e P d50f07af87d74f039fb2b43cf166fa43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:55:42.090467 28387 raft_consensus.cc:504] T f22c0dfa05664a4a9f10aa81424a396e P d50f07af87d74f039fb2b43cf166fa43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } }
I20240502 01:55:42.090498 27715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63136d6964084523bb950cc6568a273b" candidate_uuid: "a64152fd9595478ca767a9f248ef3b85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d50f07af87d74f039fb2b43cf166fa43"
I20240502 01:55:42.090677 27715 raft_consensus.cc:3037] T 63136d6964084523bb950cc6568a273b P d50f07af87d74f039fb2b43cf166fa43 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:42.091070 28387 leader_election.cc:288] T f22c0dfa05664a4a9f10aa81424a396e P d50f07af87d74f039fb2b43cf166fa43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64152fd9595478ca767a9f248ef3b85 (127.9.176.194:38689), b550bed5a64f466c8ba5a1761f3bef39 (127.9.176.195:40013)
I20240502 01:55:42.091538 27849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f22c0dfa05664a4a9f10aa81424a396e" candidate_uuid: "d50f07af87d74f039fb2b43cf166fa43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64152fd9595478ca767a9f248ef3b85" is_pre_election: true
I20240502 01:55:42.091822 27849 raft_consensus.cc:2454] T f22c0dfa05664a4a9f10aa81424a396e P a64152fd9595478ca767a9f248ef3b85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d50f07af87d74f039fb2b43cf166fa43 in term 0.
I20240502 01:55:42.091836 27983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63136d6964084523bb950cc6568a273b" candidate_uuid: "a64152fd9595478ca767a9f248ef3b85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b550bed5a64f466c8ba5a1761f3bef39"
I20240502 01:55:42.092079 27983 raft_consensus.cc:3037] T 63136d6964084523bb950cc6568a273b P b550bed5a64f466c8ba5a1761f3bef39 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:42.092435 27651 leader_election.cc:302] T f22c0dfa05664a4a9f10aa81424a396e P d50f07af87d74f039fb2b43cf166fa43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64152fd9595478ca767a9f248ef3b85, d50f07af87d74f039fb2b43cf166fa43; no voters: 
I20240502 01:55:42.092777 28387 raft_consensus.cc:2781] T f22c0dfa05664a4a9f10aa81424a396e P d50f07af87d74f039fb2b43cf166fa43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:55:42.092911 28387 raft_consensus.cc:482] T f22c0dfa05664a4a9f10aa81424a396e P d50f07af87d74f039fb2b43cf166fa43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:55:42.092998 28387 raft_consensus.cc:3037] T f22c0dfa05664a4a9f10aa81424a396e P d50f07af87d74f039fb2b43cf166fa43 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:42.093284 27715 raft_consensus.cc:2454] T 63136d6964084523bb950cc6568a273b P d50f07af87d74f039fb2b43cf166fa43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64152fd9595478ca767a9f248ef3b85 in term 1.
I20240502 01:55:42.094115 27981 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f22c0dfa05664a4a9f10aa81424a396e" candidate_uuid: "d50f07af87d74f039fb2b43cf166fa43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b550bed5a64f466c8ba5a1761f3bef39" is_pre_election: true
I20240502 01:55:42.094400 27981 raft_consensus.cc:2454] T f22c0dfa05664a4a9f10aa81424a396e P b550bed5a64f466c8ba5a1761f3bef39 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d50f07af87d74f039fb2b43cf166fa43 in term 0.
I20240502 01:55:42.094089 27982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e8b27dca9af473188c50269820343f1" candidate_uuid: "a64152fd9595478ca767a9f248ef3b85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b550bed5a64f466c8ba5a1761f3bef39"
I20240502 01:55:42.094501 27716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e8b27dca9af473188c50269820343f1" candidate_uuid: "a64152fd9595478ca767a9f248ef3b85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d50f07af87d74f039fb2b43cf166fa43"
I20240502 01:55:42.094704 27982 raft_consensus.cc:3037] T 5e8b27dca9af473188c50269820343f1 P b550bed5a64f466c8ba5a1761f3bef39 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:42.094729 27716 raft_consensus.cc:3037] T 5e8b27dca9af473188c50269820343f1 P d50f07af87d74f039fb2b43cf166fa43 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:42.095175 27983 raft_consensus.cc:2454] T 63136d6964084523bb950cc6568a273b P b550bed5a64f466c8ba5a1761f3bef39 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64152fd9595478ca767a9f248ef3b85 in term 1.
I20240502 01:55:42.095893 28387 raft_consensus.cc:504] T f22c0dfa05664a4a9f10aa81424a396e P d50f07af87d74f039fb2b43cf166fa43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } }
I20240502 01:55:42.096549 28387 leader_election.cc:288] T f22c0dfa05664a4a9f10aa81424a396e P d50f07af87d74f039fb2b43cf166fa43 [CANDIDATE]: Term 1 election: Requested vote from peers a64152fd9595478ca767a9f248ef3b85 (127.9.176.194:38689), b550bed5a64f466c8ba5a1761f3bef39 (127.9.176.195:40013)
I20240502 01:55:42.093849 27783 leader_election.cc:302] T 63136d6964084523bb950cc6568a273b P a64152fd9595478ca767a9f248ef3b85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64152fd9595478ca767a9f248ef3b85, d50f07af87d74f039fb2b43cf166fa43; no voters: 
I20240502 01:55:42.097570 28326 raft_consensus.cc:2781] T 63136d6964084523bb950cc6568a273b P a64152fd9595478ca767a9f248ef3b85 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:55:42.097956 27982 raft_consensus.cc:2454] T 5e8b27dca9af473188c50269820343f1 P b550bed5a64f466c8ba5a1761f3bef39 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64152fd9595478ca767a9f248ef3b85 in term 1.
I20240502 01:55:42.097955 27716 raft_consensus.cc:2454] T 5e8b27dca9af473188c50269820343f1 P d50f07af87d74f039fb2b43cf166fa43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64152fd9595478ca767a9f248ef3b85 in term 1.
I20240502 01:55:42.097954 27849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f22c0dfa05664a4a9f10aa81424a396e" candidate_uuid: "d50f07af87d74f039fb2b43cf166fa43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64152fd9595478ca767a9f248ef3b85"
I20240502 01:55:42.098197 27849 raft_consensus.cc:3037] T f22c0dfa05664a4a9f10aa81424a396e P a64152fd9595478ca767a9f248ef3b85 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:42.098551 27783 leader_election.cc:302] T 5e8b27dca9af473188c50269820343f1 P a64152fd9595478ca767a9f248ef3b85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: a64152fd9595478ca767a9f248ef3b85, b550bed5a64f466c8ba5a1761f3bef39, d50f07af87d74f039fb2b43cf166fa43; no voters: 
I20240502 01:55:42.098908 28326 raft_consensus.cc:686] T 63136d6964084523bb950cc6568a273b P a64152fd9595478ca767a9f248ef3b85 [term 1 LEADER]: Becoming Leader. State: Replica: a64152fd9595478ca767a9f248ef3b85, State: Running, Role: LEADER
I20240502 01:55:42.098934 27983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f22c0dfa05664a4a9f10aa81424a396e" candidate_uuid: "d50f07af87d74f039fb2b43cf166fa43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b550bed5a64f466c8ba5a1761f3bef39"
I20240502 01:55:42.099175 27983 raft_consensus.cc:3037] T f22c0dfa05664a4a9f10aa81424a396e P b550bed5a64f466c8ba5a1761f3bef39 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:42.099246 28326 consensus_queue.cc:237] T 63136d6964084523bb950cc6568a273b P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } }
I20240502 01:55:42.100088 28338 raft_consensus.cc:2781] T 5e8b27dca9af473188c50269820343f1 P a64152fd9595478ca767a9f248ef3b85 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:55:42.100416 28338 raft_consensus.cc:686] T 5e8b27dca9af473188c50269820343f1 P a64152fd9595478ca767a9f248ef3b85 [term 1 LEADER]: Becoming Leader. State: Replica: a64152fd9595478ca767a9f248ef3b85, State: Running, Role: LEADER
I20240502 01:55:42.100979 28338 consensus_queue.cc:237] T 5e8b27dca9af473188c50269820343f1 P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } }
I20240502 01:55:42.102123 27849 raft_consensus.cc:2454] T f22c0dfa05664a4a9f10aa81424a396e P a64152fd9595478ca767a9f248ef3b85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d50f07af87d74f039fb2b43cf166fa43 in term 1.
I20240502 01:55:42.102229 27983 raft_consensus.cc:2454] T f22c0dfa05664a4a9f10aa81424a396e P b550bed5a64f466c8ba5a1761f3bef39 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d50f07af87d74f039fb2b43cf166fa43 in term 1.
I20240502 01:55:42.102828 27651 leader_election.cc:302] T f22c0dfa05664a4a9f10aa81424a396e P d50f07af87d74f039fb2b43cf166fa43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b550bed5a64f466c8ba5a1761f3bef39, d50f07af87d74f039fb2b43cf166fa43; no voters: 
I20240502 01:55:42.103217 28387 raft_consensus.cc:2781] T f22c0dfa05664a4a9f10aa81424a396e P d50f07af87d74f039fb2b43cf166fa43 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:55:42.103390 28387 raft_consensus.cc:686] T f22c0dfa05664a4a9f10aa81424a396e P d50f07af87d74f039fb2b43cf166fa43 [term 1 LEADER]: Becoming Leader. State: Replica: d50f07af87d74f039fb2b43cf166fa43, State: Running, Role: LEADER
I20240502 01:55:42.103783 28387 consensus_queue.cc:237] T f22c0dfa05664a4a9f10aa81424a396e P d50f07af87d74f039fb2b43cf166fa43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } }
I20240502 01:55:42.105295 27572 catalog_manager.cc:5484] T 63136d6964084523bb950cc6568a273b P a64152fd9595478ca767a9f248ef3b85 reported cstate change: term changed from 0 to 1, leader changed from <none> to a64152fd9595478ca767a9f248ef3b85 (127.9.176.194). New cstate: current_term: 1 leader_uuid: "a64152fd9595478ca767a9f248ef3b85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } health_report { overall_health: UNKNOWN } } }
I20240502 01:55:42.108201 27574 catalog_manager.cc:5484] T f22c0dfa05664a4a9f10aa81424a396e P d50f07af87d74f039fb2b43cf166fa43 reported cstate change: term changed from 0 to 1, leader changed from <none> to d50f07af87d74f039fb2b43cf166fa43 (127.9.176.193). New cstate: current_term: 1 leader_uuid: "d50f07af87d74f039fb2b43cf166fa43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } health_report { overall_health: UNKNOWN } } }
I20240502 01:55:42.110899 27572 catalog_manager.cc:5484] T 5e8b27dca9af473188c50269820343f1 P a64152fd9595478ca767a9f248ef3b85 reported cstate change: term changed from 0 to 1, leader changed from <none> to a64152fd9595478ca767a9f248ef3b85 (127.9.176.194). New cstate: current_term: 1 leader_uuid: "a64152fd9595478ca767a9f248ef3b85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } health_report { overall_health: UNKNOWN } } }
W20240502 01:55:42.506656 28426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:55:42.507099 28426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:55:42.516238 28426 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240502 01:55:42.578994 28338 consensus_queue.cc:1035] T 5e8b27dca9af473188c50269820343f1 P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:42.631481 28455 consensus_queue.cc:1035] T 5e8b27dca9af473188c50269820343f1 P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:42.631549 28452 consensus_queue.cc:1035] T 63136d6964084523bb950cc6568a273b P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:42.651155 28455 consensus_queue.cc:1035] T 63136d6964084523bb950cc6568a273b P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:42.690613 27983 raft_consensus.cc:1261] T f22c0dfa05664a4a9f10aa81424a396e P b550bed5a64f466c8ba5a1761f3bef39 [term 1 FOLLOWER]: Refusing update from remote peer d50f07af87d74f039fb2b43cf166fa43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:55:42.691452 28467 consensus_queue.cc:1035] T f22c0dfa05664a4a9f10aa81424a396e P d50f07af87d74f039fb2b43cf166fa43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:42.715437 27849 raft_consensus.cc:1261] T f22c0dfa05664a4a9f10aa81424a396e P a64152fd9595478ca767a9f248ef3b85 [term 1 FOLLOWER]: Refusing update from remote peer d50f07af87d74f039fb2b43cf166fa43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:55:42.716795 28387 consensus_queue.cc:1035] T f22c0dfa05664a4a9f10aa81424a396e P d50f07af87d74f039fb2b43cf166fa43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:42.732374 28454 mvcc.cc:205] Tried to move back new op lower bound from 7023062805250342912 to 7023062802859999232. Current Snapshot: MvccSnapshot[applied={T|T < 7023062805250342912}]
I20240502 01:55:42.820502 28473 mvcc.cc:205] Tried to move back new op lower bound from 7023062805250342912 to 7023062802859999232. Current Snapshot: MvccSnapshot[applied={T|T < 7023062805250342912}]

INSERT report
    rows total: 2000
    time total: 302.597 ms
  time per row: 0.151299 ms
I20240502 01:55:42.973796 27574 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41110:
name: "rebalance_test_table_2"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
W20240502 01:55:42.974714 27574 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 01:55:42.984936 27827 tablet_service.cc:1467] Processing CreateTablet for tablet 31de6c641c094cce894d8513b878350c (DEFAULT_TABLE table=rebalance_test_table_2 [id=d490226c88e64345ad42c7ceb86bf959]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:42.985132 27695 tablet_service.cc:1467] Processing CreateTablet for tablet 31de6c641c094cce894d8513b878350c (DEFAULT_TABLE table=rebalance_test_table_2 [id=d490226c88e64345ad42c7ceb86bf959]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:42.985534 27827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31de6c641c094cce894d8513b878350c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:42.985687 27695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31de6c641c094cce894d8513b878350c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:42.987193 27694 tablet_service.cc:1467] Processing CreateTablet for tablet d404c85b11594e7eb1257fcf384beafa (DEFAULT_TABLE table=rebalance_test_table_2 [id=d490226c88e64345ad42c7ceb86bf959]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:42.987589 27961 tablet_service.cc:1467] Processing CreateTablet for tablet 31de6c641c094cce894d8513b878350c (DEFAULT_TABLE table=rebalance_test_table_2 [id=d490226c88e64345ad42c7ceb86bf959]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:42.987805 27694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d404c85b11594e7eb1257fcf384beafa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:42.988078 27961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31de6c641c094cce894d8513b878350c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:42.992218 27696 tablet_service.cc:1467] Processing CreateTablet for tablet b89e275e1e274042a2cc120111080433 (DEFAULT_TABLE table=rebalance_test_table_2 [id=d490226c88e64345ad42c7ceb86bf959]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:42.992787 27696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b89e275e1e274042a2cc120111080433. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:42.993886 27960 tablet_service.cc:1467] Processing CreateTablet for tablet d404c85b11594e7eb1257fcf384beafa (DEFAULT_TABLE table=rebalance_test_table_2 [id=d490226c88e64345ad42c7ceb86bf959]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:42.994488 27960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d404c85b11594e7eb1257fcf384beafa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:43.000276 27962 tablet_service.cc:1467] Processing CreateTablet for tablet b89e275e1e274042a2cc120111080433 (DEFAULT_TABLE table=rebalance_test_table_2 [id=d490226c88e64345ad42c7ceb86bf959]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:43.001005 27962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b89e275e1e274042a2cc120111080433. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:43.004981 27828 tablet_service.cc:1467] Processing CreateTablet for tablet d404c85b11594e7eb1257fcf384beafa (DEFAULT_TABLE table=rebalance_test_table_2 [id=d490226c88e64345ad42c7ceb86bf959]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:43.005841 27828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d404c85b11594e7eb1257fcf384beafa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:43.008350 27829 tablet_service.cc:1467] Processing CreateTablet for tablet b89e275e1e274042a2cc120111080433 (DEFAULT_TABLE table=rebalance_test_table_2 [id=d490226c88e64345ad42c7ceb86bf959]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:43.009074 27829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b89e275e1e274042a2cc120111080433. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:43.012697 28494 tablet_bootstrap.cc:492] T b89e275e1e274042a2cc120111080433 P b550bed5a64f466c8ba5a1761f3bef39: Bootstrap starting.
I20240502 01:55:43.014827 28494 tablet_bootstrap.cc:654] T b89e275e1e274042a2cc120111080433 P b550bed5a64f466c8ba5a1761f3bef39: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:43.020505 28493 tablet_bootstrap.cc:492] T 31de6c641c094cce894d8513b878350c P d50f07af87d74f039fb2b43cf166fa43: Bootstrap starting.
I20240502 01:55:43.022061 28494 tablet_bootstrap.cc:492] T b89e275e1e274042a2cc120111080433 P b550bed5a64f466c8ba5a1761f3bef39: No bootstrap required, opened a new log
I20240502 01:55:43.022284 28494 ts_tablet_manager.cc:1367] T b89e275e1e274042a2cc120111080433 P b550bed5a64f466c8ba5a1761f3bef39: Time spent bootstrapping tablet: real 0.010s	user 0.000s	sys 0.004s
I20240502 01:55:43.022557 28493 tablet_bootstrap.cc:654] T 31de6c641c094cce894d8513b878350c P d50f07af87d74f039fb2b43cf166fa43: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:43.023294 28494 raft_consensus.cc:348] T b89e275e1e274042a2cc120111080433 P b550bed5a64f466c8ba5a1761f3bef39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } }
I20240502 01:55:43.023531 28494 raft_consensus.cc:374] T b89e275e1e274042a2cc120111080433 P b550bed5a64f466c8ba5a1761f3bef39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:43.023687 28494 raft_consensus.cc:729] T b89e275e1e274042a2cc120111080433 P b550bed5a64f466c8ba5a1761f3bef39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b550bed5a64f466c8ba5a1761f3bef39, State: Initialized, Role: FOLLOWER
I20240502 01:55:43.024052 28494 consensus_queue.cc:260] T b89e275e1e274042a2cc120111080433 P b550bed5a64f466c8ba5a1761f3bef39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } }
I20240502 01:55:43.024874 28494 ts_tablet_manager.cc:1398] T b89e275e1e274042a2cc120111080433 P b550bed5a64f466c8ba5a1761f3bef39: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 01:55:43.025327 28494 tablet_bootstrap.cc:492] T 31de6c641c094cce894d8513b878350c P b550bed5a64f466c8ba5a1761f3bef39: Bootstrap starting.
I20240502 01:55:43.027238 28494 tablet_bootstrap.cc:654] T 31de6c641c094cce894d8513b878350c P b550bed5a64f466c8ba5a1761f3bef39: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:43.036618 28494 tablet_bootstrap.cc:492] T 31de6c641c094cce894d8513b878350c P b550bed5a64f466c8ba5a1761f3bef39: No bootstrap required, opened a new log
I20240502 01:55:43.036919 28494 ts_tablet_manager.cc:1367] T 31de6c641c094cce894d8513b878350c P b550bed5a64f466c8ba5a1761f3bef39: Time spent bootstrapping tablet: real 0.012s	user 0.005s	sys 0.005s
I20240502 01:55:43.038244 28494 raft_consensus.cc:348] T 31de6c641c094cce894d8513b878350c P b550bed5a64f466c8ba5a1761f3bef39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } }
I20240502 01:55:43.038611 28494 raft_consensus.cc:374] T 31de6c641c094cce894d8513b878350c P b550bed5a64f466c8ba5a1761f3bef39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:43.038703 28494 raft_consensus.cc:729] T 31de6c641c094cce894d8513b878350c P b550bed5a64f466c8ba5a1761f3bef39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b550bed5a64f466c8ba5a1761f3bef39, State: Initialized, Role: FOLLOWER
I20240502 01:55:43.039124 28494 consensus_queue.cc:260] T 31de6c641c094cce894d8513b878350c P b550bed5a64f466c8ba5a1761f3bef39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } }
I20240502 01:55:43.040156 28494 ts_tablet_manager.cc:1398] T 31de6c641c094cce894d8513b878350c P b550bed5a64f466c8ba5a1761f3bef39: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 01:55:43.041168 28494 tablet_bootstrap.cc:492] T d404c85b11594e7eb1257fcf384beafa P b550bed5a64f466c8ba5a1761f3bef39: Bootstrap starting.
I20240502 01:55:43.043133 28494 tablet_bootstrap.cc:654] T d404c85b11594e7eb1257fcf384beafa P b550bed5a64f466c8ba5a1761f3bef39: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:43.058176 28492 tablet_bootstrap.cc:492] T 31de6c641c094cce894d8513b878350c P a64152fd9595478ca767a9f248ef3b85: Bootstrap starting.
I20240502 01:55:43.060030 28494 tablet_bootstrap.cc:492] T d404c85b11594e7eb1257fcf384beafa P b550bed5a64f466c8ba5a1761f3bef39: No bootstrap required, opened a new log
I20240502 01:55:43.060143 28493 tablet_bootstrap.cc:492] T 31de6c641c094cce894d8513b878350c P d50f07af87d74f039fb2b43cf166fa43: No bootstrap required, opened a new log
I20240502 01:55:43.060225 28494 ts_tablet_manager.cc:1367] T d404c85b11594e7eb1257fcf384beafa P b550bed5a64f466c8ba5a1761f3bef39: Time spent bootstrapping tablet: real 0.019s	user 0.004s	sys 0.014s
I20240502 01:55:43.060379 28493 ts_tablet_manager.cc:1367] T 31de6c641c094cce894d8513b878350c P d50f07af87d74f039fb2b43cf166fa43: Time spent bootstrapping tablet: real 0.040s	user 0.015s	sys 0.015s
I20240502 01:55:43.061003 28494 raft_consensus.cc:348] T d404c85b11594e7eb1257fcf384beafa P b550bed5a64f466c8ba5a1761f3bef39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } }
I20240502 01:55:43.061234 28494 raft_consensus.cc:374] T d404c85b11594e7eb1257fcf384beafa P b550bed5a64f466c8ba5a1761f3bef39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:43.061347 28494 raft_consensus.cc:729] T d404c85b11594e7eb1257fcf384beafa P b550bed5a64f466c8ba5a1761f3bef39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b550bed5a64f466c8ba5a1761f3bef39, State: Initialized, Role: FOLLOWER
I20240502 01:55:43.061487 28492 tablet_bootstrap.cc:654] T 31de6c641c094cce894d8513b878350c P a64152fd9595478ca767a9f248ef3b85: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:43.061380 28493 raft_consensus.cc:348] T 31de6c641c094cce894d8513b878350c P d50f07af87d74f039fb2b43cf166fa43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } }
I20240502 01:55:43.061642 28493 raft_consensus.cc:374] T 31de6c641c094cce894d8513b878350c P d50f07af87d74f039fb2b43cf166fa43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:43.061743 28493 raft_consensus.cc:729] T 31de6c641c094cce894d8513b878350c P d50f07af87d74f039fb2b43cf166fa43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d50f07af87d74f039fb2b43cf166fa43, State: Initialized, Role: FOLLOWER
I20240502 01:55:43.061654 28494 consensus_queue.cc:260] T d404c85b11594e7eb1257fcf384beafa P b550bed5a64f466c8ba5a1761f3bef39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } }
I20240502 01:55:43.062037 28493 consensus_queue.cc:260] T 31de6c641c094cce894d8513b878350c P d50f07af87d74f039fb2b43cf166fa43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } }
I20240502 01:55:43.062649 28494 ts_tablet_manager.cc:1398] T d404c85b11594e7eb1257fcf384beafa P b550bed5a64f466c8ba5a1761f3bef39: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.003s
I20240502 01:55:43.062747 28493 ts_tablet_manager.cc:1398] T 31de6c641c094cce894d8513b878350c P d50f07af87d74f039fb2b43cf166fa43: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 01:55:43.063066 28493 tablet_bootstrap.cc:492] T d404c85b11594e7eb1257fcf384beafa P d50f07af87d74f039fb2b43cf166fa43: Bootstrap starting.
I20240502 01:55:43.065418 28493 tablet_bootstrap.cc:654] T d404c85b11594e7eb1257fcf384beafa P d50f07af87d74f039fb2b43cf166fa43: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:43.066112 28492 tablet_bootstrap.cc:492] T 31de6c641c094cce894d8513b878350c P a64152fd9595478ca767a9f248ef3b85: No bootstrap required, opened a new log
I20240502 01:55:43.066296 28492 ts_tablet_manager.cc:1367] T 31de6c641c094cce894d8513b878350c P a64152fd9595478ca767a9f248ef3b85: Time spent bootstrapping tablet: real 0.008s	user 0.005s	sys 0.000s
I20240502 01:55:43.067325 28492 raft_consensus.cc:348] T 31de6c641c094cce894d8513b878350c P a64152fd9595478ca767a9f248ef3b85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } }
I20240502 01:55:43.067656 28492 raft_consensus.cc:374] T 31de6c641c094cce894d8513b878350c P a64152fd9595478ca767a9f248ef3b85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:43.067803 28492 raft_consensus.cc:729] T 31de6c641c094cce894d8513b878350c P a64152fd9595478ca767a9f248ef3b85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64152fd9595478ca767a9f248ef3b85, State: Initialized, Role: FOLLOWER
I20240502 01:55:43.068076 28492 consensus_queue.cc:260] T 31de6c641c094cce894d8513b878350c P a64152fd9595478ca767a9f248ef3b85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } }
I20240502 01:55:43.068457 28493 tablet_bootstrap.cc:492] T d404c85b11594e7eb1257fcf384beafa P d50f07af87d74f039fb2b43cf166fa43: No bootstrap required, opened a new log
I20240502 01:55:43.068670 28493 ts_tablet_manager.cc:1367] T d404c85b11594e7eb1257fcf384beafa P d50f07af87d74f039fb2b43cf166fa43: Time spent bootstrapping tablet: real 0.006s	user 0.004s	sys 0.000s
I20240502 01:55:43.068861 28492 ts_tablet_manager.cc:1398] T 31de6c641c094cce894d8513b878350c P a64152fd9595478ca767a9f248ef3b85: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.000s
I20240502 01:55:43.069185 28492 tablet_bootstrap.cc:492] T b89e275e1e274042a2cc120111080433 P a64152fd9595478ca767a9f248ef3b85: Bootstrap starting.
I20240502 01:55:43.069921 28493 raft_consensus.cc:348] T d404c85b11594e7eb1257fcf384beafa P d50f07af87d74f039fb2b43cf166fa43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } }
I20240502 01:55:43.070235 28493 raft_consensus.cc:374] T d404c85b11594e7eb1257fcf384beafa P d50f07af87d74f039fb2b43cf166fa43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:43.070341 28493 raft_consensus.cc:729] T d404c85b11594e7eb1257fcf384beafa P d50f07af87d74f039fb2b43cf166fa43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d50f07af87d74f039fb2b43cf166fa43, State: Initialized, Role: FOLLOWER
I20240502 01:55:43.070672 28493 consensus_queue.cc:260] T d404c85b11594e7eb1257fcf384beafa P d50f07af87d74f039fb2b43cf166fa43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } }
I20240502 01:55:43.071072 28492 tablet_bootstrap.cc:654] T b89e275e1e274042a2cc120111080433 P a64152fd9595478ca767a9f248ef3b85: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:43.071498 28493 ts_tablet_manager.cc:1398] T d404c85b11594e7eb1257fcf384beafa P d50f07af87d74f039fb2b43cf166fa43: Time spent starting tablet: real 0.003s	user 0.003s	sys 0.000s
I20240502 01:55:43.071928 28493 tablet_bootstrap.cc:492] T b89e275e1e274042a2cc120111080433 P d50f07af87d74f039fb2b43cf166fa43: Bootstrap starting.
I20240502 01:55:43.073956 28493 tablet_bootstrap.cc:654] T b89e275e1e274042a2cc120111080433 P d50f07af87d74f039fb2b43cf166fa43: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:43.076511 28493 tablet_bootstrap.cc:492] T b89e275e1e274042a2cc120111080433 P d50f07af87d74f039fb2b43cf166fa43: No bootstrap required, opened a new log
I20240502 01:55:43.076754 28493 ts_tablet_manager.cc:1367] T b89e275e1e274042a2cc120111080433 P d50f07af87d74f039fb2b43cf166fa43: Time spent bootstrapping tablet: real 0.005s	user 0.003s	sys 0.000s
I20240502 01:55:43.077260 28492 tablet_bootstrap.cc:492] T b89e275e1e274042a2cc120111080433 P a64152fd9595478ca767a9f248ef3b85: No bootstrap required, opened a new log
I20240502 01:55:43.077492 28492 ts_tablet_manager.cc:1367] T b89e275e1e274042a2cc120111080433 P a64152fd9595478ca767a9f248ef3b85: Time spent bootstrapping tablet: real 0.008s	user 0.006s	sys 0.000s
I20240502 01:55:43.077788 28493 raft_consensus.cc:348] T b89e275e1e274042a2cc120111080433 P d50f07af87d74f039fb2b43cf166fa43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } }
I20240502 01:55:43.078043 28493 raft_consensus.cc:374] T b89e275e1e274042a2cc120111080433 P d50f07af87d74f039fb2b43cf166fa43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:43.078162 28493 raft_consensus.cc:729] T b89e275e1e274042a2cc120111080433 P d50f07af87d74f039fb2b43cf166fa43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d50f07af87d74f039fb2b43cf166fa43, State: Initialized, Role: FOLLOWER
I20240502 01:55:43.078459 28493 consensus_queue.cc:260] T b89e275e1e274042a2cc120111080433 P d50f07af87d74f039fb2b43cf166fa43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } }
I20240502 01:55:43.078531 28492 raft_consensus.cc:348] T b89e275e1e274042a2cc120111080433 P a64152fd9595478ca767a9f248ef3b85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } }
I20240502 01:55:43.078781 28492 raft_consensus.cc:374] T b89e275e1e274042a2cc120111080433 P a64152fd9595478ca767a9f248ef3b85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:43.078878 28492 raft_consensus.cc:729] T b89e275e1e274042a2cc120111080433 P a64152fd9595478ca767a9f248ef3b85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64152fd9595478ca767a9f248ef3b85, State: Initialized, Role: FOLLOWER
I20240502 01:55:43.079181 28492 consensus_queue.cc:260] T b89e275e1e274042a2cc120111080433 P a64152fd9595478ca767a9f248ef3b85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } }
I20240502 01:55:43.079900 28492 ts_tablet_manager.cc:1398] T b89e275e1e274042a2cc120111080433 P a64152fd9595478ca767a9f248ef3b85: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.000s
I20240502 01:55:43.080271 28492 tablet_bootstrap.cc:492] T d404c85b11594e7eb1257fcf384beafa P a64152fd9595478ca767a9f248ef3b85: Bootstrap starting.
I20240502 01:55:43.081024 28493 ts_tablet_manager.cc:1398] T b89e275e1e274042a2cc120111080433 P d50f07af87d74f039fb2b43cf166fa43: Time spent starting tablet: real 0.004s	user 0.003s	sys 0.001s
I20240502 01:55:43.082110 28492 tablet_bootstrap.cc:654] T d404c85b11594e7eb1257fcf384beafa P a64152fd9595478ca767a9f248ef3b85: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:43.084652 28492 tablet_bootstrap.cc:492] T d404c85b11594e7eb1257fcf384beafa P a64152fd9595478ca767a9f248ef3b85: No bootstrap required, opened a new log
I20240502 01:55:43.084875 28492 ts_tablet_manager.cc:1367] T d404c85b11594e7eb1257fcf384beafa P a64152fd9595478ca767a9f248ef3b85: Time spent bootstrapping tablet: real 0.005s	user 0.004s	sys 0.000s
I20240502 01:55:43.086020 28492 raft_consensus.cc:348] T d404c85b11594e7eb1257fcf384beafa P a64152fd9595478ca767a9f248ef3b85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } }
I20240502 01:55:43.086225 28492 raft_consensus.cc:374] T d404c85b11594e7eb1257fcf384beafa P a64152fd9595478ca767a9f248ef3b85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:43.086330 28492 raft_consensus.cc:729] T d404c85b11594e7eb1257fcf384beafa P a64152fd9595478ca767a9f248ef3b85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64152fd9595478ca767a9f248ef3b85, State: Initialized, Role: FOLLOWER
I20240502 01:55:43.086586 28492 consensus_queue.cc:260] T d404c85b11594e7eb1257fcf384beafa P a64152fd9595478ca767a9f248ef3b85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } }
I20240502 01:55:43.087191 28492 ts_tablet_manager.cc:1398] T d404c85b11594e7eb1257fcf384beafa P a64152fd9595478ca767a9f248ef3b85: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240502 01:55:43.183773 28455 raft_consensus.cc:482] T 31de6c641c094cce894d8513b878350c P a64152fd9595478ca767a9f248ef3b85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:55:43.184028 28455 raft_consensus.cc:504] T 31de6c641c094cce894d8513b878350c P a64152fd9595478ca767a9f248ef3b85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } }
I20240502 01:55:43.184878 28455 leader_election.cc:288] T 31de6c641c094cce894d8513b878350c P a64152fd9595478ca767a9f248ef3b85 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d50f07af87d74f039fb2b43cf166fa43 (127.9.176.193:34203), b550bed5a64f466c8ba5a1761f3bef39 (127.9.176.195:40013)
I20240502 01:55:43.185483 27714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31de6c641c094cce894d8513b878350c" candidate_uuid: "a64152fd9595478ca767a9f248ef3b85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d50f07af87d74f039fb2b43cf166fa43" is_pre_election: true
I20240502 01:55:43.185508 27981 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31de6c641c094cce894d8513b878350c" candidate_uuid: "a64152fd9595478ca767a9f248ef3b85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b550bed5a64f466c8ba5a1761f3bef39" is_pre_election: true
I20240502 01:55:43.185724 27714 raft_consensus.cc:2454] T 31de6c641c094cce894d8513b878350c P d50f07af87d74f039fb2b43cf166fa43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64152fd9595478ca767a9f248ef3b85 in term 0.
I20240502 01:55:43.185729 27981 raft_consensus.cc:2454] T 31de6c641c094cce894d8513b878350c P b550bed5a64f466c8ba5a1761f3bef39 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64152fd9595478ca767a9f248ef3b85 in term 0.
I20240502 01:55:43.186203 27783 leader_election.cc:302] T 31de6c641c094cce894d8513b878350c P a64152fd9595478ca767a9f248ef3b85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64152fd9595478ca767a9f248ef3b85, d50f07af87d74f039fb2b43cf166fa43; no voters: 
I20240502 01:55:43.186538 28455 raft_consensus.cc:2781] T 31de6c641c094cce894d8513b878350c P a64152fd9595478ca767a9f248ef3b85 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:55:43.186682 28455 raft_consensus.cc:482] T 31de6c641c094cce894d8513b878350c P a64152fd9595478ca767a9f248ef3b85 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:55:43.186782 28455 raft_consensus.cc:3037] T 31de6c641c094cce894d8513b878350c P a64152fd9595478ca767a9f248ef3b85 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:43.188959 28455 raft_consensus.cc:504] T 31de6c641c094cce894d8513b878350c P a64152fd9595478ca767a9f248ef3b85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } }
I20240502 01:55:43.189792 28455 leader_election.cc:288] T 31de6c641c094cce894d8513b878350c P a64152fd9595478ca767a9f248ef3b85 [CANDIDATE]: Term 1 election: Requested vote from peers d50f07af87d74f039fb2b43cf166fa43 (127.9.176.193:34203), b550bed5a64f466c8ba5a1761f3bef39 (127.9.176.195:40013)
I20240502 01:55:43.190304 27714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31de6c641c094cce894d8513b878350c" candidate_uuid: "a64152fd9595478ca767a9f248ef3b85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d50f07af87d74f039fb2b43cf166fa43"
I20240502 01:55:43.190387 27981 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31de6c641c094cce894d8513b878350c" candidate_uuid: "a64152fd9595478ca767a9f248ef3b85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b550bed5a64f466c8ba5a1761f3bef39"
I20240502 01:55:43.190521 27714 raft_consensus.cc:3037] T 31de6c641c094cce894d8513b878350c P d50f07af87d74f039fb2b43cf166fa43 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:43.190670 27981 raft_consensus.cc:3037] T 31de6c641c094cce894d8513b878350c P b550bed5a64f466c8ba5a1761f3bef39 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:43.192735 27714 raft_consensus.cc:2454] T 31de6c641c094cce894d8513b878350c P d50f07af87d74f039fb2b43cf166fa43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64152fd9595478ca767a9f248ef3b85 in term 1.
I20240502 01:55:43.192965 27981 raft_consensus.cc:2454] T 31de6c641c094cce894d8513b878350c P b550bed5a64f466c8ba5a1761f3bef39 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64152fd9595478ca767a9f248ef3b85 in term 1.
I20240502 01:55:43.193401 27783 leader_election.cc:302] T 31de6c641c094cce894d8513b878350c P a64152fd9595478ca767a9f248ef3b85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64152fd9595478ca767a9f248ef3b85, d50f07af87d74f039fb2b43cf166fa43; no voters: 
I20240502 01:55:43.193750 28455 raft_consensus.cc:2781] T 31de6c641c094cce894d8513b878350c P a64152fd9595478ca767a9f248ef3b85 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:55:43.193898 28455 raft_consensus.cc:686] T 31de6c641c094cce894d8513b878350c P a64152fd9595478ca767a9f248ef3b85 [term 1 LEADER]: Becoming Leader. State: Replica: a64152fd9595478ca767a9f248ef3b85, State: Running, Role: LEADER
I20240502 01:55:43.194226 28455 consensus_queue.cc:237] T 31de6c641c094cce894d8513b878350c P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } }
I20240502 01:55:43.197342 27567 catalog_manager.cc:5484] T 31de6c641c094cce894d8513b878350c P a64152fd9595478ca767a9f248ef3b85 reported cstate change: term changed from 0 to 1, leader changed from <none> to a64152fd9595478ca767a9f248ef3b85 (127.9.176.194). New cstate: current_term: 1 leader_uuid: "a64152fd9595478ca767a9f248ef3b85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } health_report { overall_health: UNKNOWN } } }
I20240502 01:55:43.249229 28455 raft_consensus.cc:482] T d404c85b11594e7eb1257fcf384beafa P a64152fd9595478ca767a9f248ef3b85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:55:43.249447 28455 raft_consensus.cc:504] T d404c85b11594e7eb1257fcf384beafa P a64152fd9595478ca767a9f248ef3b85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } }
I20240502 01:55:43.250106 28455 leader_election.cc:288] T d404c85b11594e7eb1257fcf384beafa P a64152fd9595478ca767a9f248ef3b85 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d50f07af87d74f039fb2b43cf166fa43 (127.9.176.193:34203), b550bed5a64f466c8ba5a1761f3bef39 (127.9.176.195:40013)
I20240502 01:55:43.250643 27714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d404c85b11594e7eb1257fcf384beafa" candidate_uuid: "a64152fd9595478ca767a9f248ef3b85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d50f07af87d74f039fb2b43cf166fa43" is_pre_election: true
I20240502 01:55:43.250648 27981 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d404c85b11594e7eb1257fcf384beafa" candidate_uuid: "a64152fd9595478ca767a9f248ef3b85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b550bed5a64f466c8ba5a1761f3bef39" is_pre_election: true
I20240502 01:55:43.250872 27714 raft_consensus.cc:2454] T d404c85b11594e7eb1257fcf384beafa P d50f07af87d74f039fb2b43cf166fa43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64152fd9595478ca767a9f248ef3b85 in term 0.
I20240502 01:55:43.250890 27981 raft_consensus.cc:2454] T d404c85b11594e7eb1257fcf384beafa P b550bed5a64f466c8ba5a1761f3bef39 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64152fd9595478ca767a9f248ef3b85 in term 0.
I20240502 01:55:43.251305 27783 leader_election.cc:302] T d404c85b11594e7eb1257fcf384beafa P a64152fd9595478ca767a9f248ef3b85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64152fd9595478ca767a9f248ef3b85, d50f07af87d74f039fb2b43cf166fa43; no voters: 
I20240502 01:55:43.251703 28455 raft_consensus.cc:2781] T d404c85b11594e7eb1257fcf384beafa P a64152fd9595478ca767a9f248ef3b85 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:55:43.251806 28455 raft_consensus.cc:482] T d404c85b11594e7eb1257fcf384beafa P a64152fd9595478ca767a9f248ef3b85 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:55:43.251917 28455 raft_consensus.cc:3037] T d404c85b11594e7eb1257fcf384beafa P a64152fd9595478ca767a9f248ef3b85 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:43.253980 28455 raft_consensus.cc:504] T d404c85b11594e7eb1257fcf384beafa P a64152fd9595478ca767a9f248ef3b85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } }
I20240502 01:55:43.254673 28455 leader_election.cc:288] T d404c85b11594e7eb1257fcf384beafa P a64152fd9595478ca767a9f248ef3b85 [CANDIDATE]: Term 1 election: Requested vote from peers d50f07af87d74f039fb2b43cf166fa43 (127.9.176.193:34203), b550bed5a64f466c8ba5a1761f3bef39 (127.9.176.195:40013)
I20240502 01:55:43.255224 27714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d404c85b11594e7eb1257fcf384beafa" candidate_uuid: "a64152fd9595478ca767a9f248ef3b85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d50f07af87d74f039fb2b43cf166fa43"
I20240502 01:55:43.255455 27714 raft_consensus.cc:3037] T d404c85b11594e7eb1257fcf384beafa P d50f07af87d74f039fb2b43cf166fa43 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:43.255798 27981 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d404c85b11594e7eb1257fcf384beafa" candidate_uuid: "a64152fd9595478ca767a9f248ef3b85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b550bed5a64f466c8ba5a1761f3bef39"
I20240502 01:55:43.256019 27981 raft_consensus.cc:3037] T d404c85b11594e7eb1257fcf384beafa P b550bed5a64f466c8ba5a1761f3bef39 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:43.257516 27714 raft_consensus.cc:2454] T d404c85b11594e7eb1257fcf384beafa P d50f07af87d74f039fb2b43cf166fa43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64152fd9595478ca767a9f248ef3b85 in term 1.
I20240502 01:55:43.258101 27783 leader_election.cc:302] T d404c85b11594e7eb1257fcf384beafa P a64152fd9595478ca767a9f248ef3b85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64152fd9595478ca767a9f248ef3b85, d50f07af87d74f039fb2b43cf166fa43; no voters: 
I20240502 01:55:43.258278 27981 raft_consensus.cc:2454] T d404c85b11594e7eb1257fcf384beafa P b550bed5a64f466c8ba5a1761f3bef39 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64152fd9595478ca767a9f248ef3b85 in term 1.
I20240502 01:55:43.258513 28455 raft_consensus.cc:2781] T d404c85b11594e7eb1257fcf384beafa P a64152fd9595478ca767a9f248ef3b85 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:55:43.258821 28455 raft_consensus.cc:686] T d404c85b11594e7eb1257fcf384beafa P a64152fd9595478ca767a9f248ef3b85 [term 1 LEADER]: Becoming Leader. State: Replica: a64152fd9595478ca767a9f248ef3b85, State: Running, Role: LEADER
I20240502 01:55:43.259394 28455 consensus_queue.cc:237] T d404c85b11594e7eb1257fcf384beafa P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } }
I20240502 01:55:43.262503 27567 catalog_manager.cc:5484] T d404c85b11594e7eb1257fcf384beafa P a64152fd9595478ca767a9f248ef3b85 reported cstate change: term changed from 0 to 1, leader changed from <none> to a64152fd9595478ca767a9f248ef3b85 (127.9.176.194). New cstate: current_term: 1 leader_uuid: "a64152fd9595478ca767a9f248ef3b85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } health_report { overall_health: UNKNOWN } } }
I20240502 01:55:43.264897 28455 raft_consensus.cc:482] T b89e275e1e274042a2cc120111080433 P a64152fd9595478ca767a9f248ef3b85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:55:43.265095 28455 raft_consensus.cc:504] T b89e275e1e274042a2cc120111080433 P a64152fd9595478ca767a9f248ef3b85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } }
I20240502 01:55:43.265822 28455 leader_election.cc:288] T b89e275e1e274042a2cc120111080433 P a64152fd9595478ca767a9f248ef3b85 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d50f07af87d74f039fb2b43cf166fa43 (127.9.176.193:34203), b550bed5a64f466c8ba5a1761f3bef39 (127.9.176.195:40013)
I20240502 01:55:43.266395 27714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b89e275e1e274042a2cc120111080433" candidate_uuid: "a64152fd9595478ca767a9f248ef3b85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d50f07af87d74f039fb2b43cf166fa43" is_pre_election: true
I20240502 01:55:43.266649 27714 raft_consensus.cc:2454] T b89e275e1e274042a2cc120111080433 P d50f07af87d74f039fb2b43cf166fa43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64152fd9595478ca767a9f248ef3b85 in term 0.
I20240502 01:55:43.266811 27981 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b89e275e1e274042a2cc120111080433" candidate_uuid: "a64152fd9595478ca767a9f248ef3b85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b550bed5a64f466c8ba5a1761f3bef39" is_pre_election: true
I20240502 01:55:43.267082 27981 raft_consensus.cc:2454] T b89e275e1e274042a2cc120111080433 P b550bed5a64f466c8ba5a1761f3bef39 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64152fd9595478ca767a9f248ef3b85 in term 0.
I20240502 01:55:43.267117 27783 leader_election.cc:302] T b89e275e1e274042a2cc120111080433 P a64152fd9595478ca767a9f248ef3b85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64152fd9595478ca767a9f248ef3b85, d50f07af87d74f039fb2b43cf166fa43; no voters: 
I20240502 01:55:43.267400 28455 raft_consensus.cc:2781] T b89e275e1e274042a2cc120111080433 P a64152fd9595478ca767a9f248ef3b85 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:55:43.267515 28455 raft_consensus.cc:482] T b89e275e1e274042a2cc120111080433 P a64152fd9595478ca767a9f248ef3b85 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:55:43.267642 28455 raft_consensus.cc:3037] T b89e275e1e274042a2cc120111080433 P a64152fd9595478ca767a9f248ef3b85 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:43.269642 28455 raft_consensus.cc:504] T b89e275e1e274042a2cc120111080433 P a64152fd9595478ca767a9f248ef3b85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } }
I20240502 01:55:43.270490 28455 leader_election.cc:288] T b89e275e1e274042a2cc120111080433 P a64152fd9595478ca767a9f248ef3b85 [CANDIDATE]: Term 1 election: Requested vote from peers d50f07af87d74f039fb2b43cf166fa43 (127.9.176.193:34203), b550bed5a64f466c8ba5a1761f3bef39 (127.9.176.195:40013)
I20240502 01:55:43.271472 27981 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b89e275e1e274042a2cc120111080433" candidate_uuid: "a64152fd9595478ca767a9f248ef3b85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b550bed5a64f466c8ba5a1761f3bef39"
I20240502 01:55:43.271473 27714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b89e275e1e274042a2cc120111080433" candidate_uuid: "a64152fd9595478ca767a9f248ef3b85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d50f07af87d74f039fb2b43cf166fa43"
I20240502 01:55:43.271783 27981 raft_consensus.cc:3037] T b89e275e1e274042a2cc120111080433 P b550bed5a64f466c8ba5a1761f3bef39 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:43.271785 27714 raft_consensus.cc:3037] T b89e275e1e274042a2cc120111080433 P d50f07af87d74f039fb2b43cf166fa43 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:43.274868 27981 raft_consensus.cc:2454] T b89e275e1e274042a2cc120111080433 P b550bed5a64f466c8ba5a1761f3bef39 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64152fd9595478ca767a9f248ef3b85 in term 1.
I20240502 01:55:43.274876 27714 raft_consensus.cc:2454] T b89e275e1e274042a2cc120111080433 P d50f07af87d74f039fb2b43cf166fa43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64152fd9595478ca767a9f248ef3b85 in term 1.
I20240502 01:55:43.275522 27784 leader_election.cc:302] T b89e275e1e274042a2cc120111080433 P a64152fd9595478ca767a9f248ef3b85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64152fd9595478ca767a9f248ef3b85, b550bed5a64f466c8ba5a1761f3bef39; no voters: 
I20240502 01:55:43.275954 28455 raft_consensus.cc:2781] T b89e275e1e274042a2cc120111080433 P a64152fd9595478ca767a9f248ef3b85 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:55:43.276142 28455 raft_consensus.cc:686] T b89e275e1e274042a2cc120111080433 P a64152fd9595478ca767a9f248ef3b85 [term 1 LEADER]: Becoming Leader. State: Replica: a64152fd9595478ca767a9f248ef3b85, State: Running, Role: LEADER
I20240502 01:55:43.276480 28455 consensus_queue.cc:237] T b89e275e1e274042a2cc120111080433 P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } }
I20240502 01:55:43.279927 27567 catalog_manager.cc:5484] T b89e275e1e274042a2cc120111080433 P a64152fd9595478ca767a9f248ef3b85 reported cstate change: term changed from 0 to 1, leader changed from <none> to a64152fd9595478ca767a9f248ef3b85 (127.9.176.194). New cstate: current_term: 1 leader_uuid: "a64152fd9595478ca767a9f248ef3b85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } health_report { overall_health: UNKNOWN } } }
I20240502 01:55:43.684593 28455 consensus_queue.cc:1035] T d404c85b11594e7eb1257fcf384beafa P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:43.686470 28455 consensus_queue.cc:1035] T 31de6c641c094cce894d8513b878350c P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:43.701467 28455 consensus_queue.cc:1035] T 31de6c641c094cce894d8513b878350c P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:43.709537 28460 consensus_queue.cc:1035] T d404c85b11594e7eb1257fcf384beafa P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:55:43.737052 28507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:55:43.737390 28507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:55:43.745183 28507 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240502 01:55:43.837606 28460 consensus_queue.cc:1035] T b89e275e1e274042a2cc120111080433 P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:43.849483 28460 consensus_queue.cc:1035] T b89e275e1e274042a2cc120111080433 P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 249.659 ms
  time per row: 0.124829 ms
I20240502 01:55:44.146860  9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
/tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.197:32773
--local_ip_for_outbound_sockets=127.9.176.197
--tserver_master_addrs=127.9.176.254:41377
--webserver_port=42969
--webserver_interface=127.9.176.197
--builtin_ntp_servers=127.9.176.212:46301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 01:55:44.522900 28563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:55:44.523247 28563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:55:44.523344 28563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:55:44.531859 28563 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:55:44.531996 28563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:55:44.532124 28563 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.197
I20240502 01:55:44.541914 28563 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:46301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.197:32773
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.9.176.197
--webserver_port=42969
--tserver_master_addrs=127.9.176.254:41377
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.197
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:55:44.543179 28563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:55:44.544934 28563 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:55:44.557617 28568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:44.558037 28569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:44.558645 28571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:55:44.559197 28563 server_base.cc:1034] running on GCE node
I20240502 01:55:44.560024 28563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:55:44.561115 28563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:55:44.562369 28563 hybrid_clock.cc:648] HybridClock initialized: now 1714614944562315 us; error 52 us; skew 500 ppm
I20240502 01:55:44.562736 28563 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:55:44.565201 28563 webserver.cc:460] Webserver started at http://127.9.176.197:42969/ using document root <none> and password file <none>
I20240502 01:55:44.566018 28563 fs_manager.cc:359] Metadata directory not provided
I20240502 01:55:44.566172 28563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:55:44.572777 28563 fs_manager.cc:709] Time spent opening directory manager: real 0.004s	user 0.003s	sys 0.001s
I20240502 01:55:44.576330 28577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:55:44.577987 28563 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.004s	sys 0.000s
I20240502 01:55:44.578224 28563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "c541378ca18944c49c37f49097bc7f81"
format_stamp: "Formatted at 2024-05-02 01:55:36 on dist-test-slave-j3hh"
I20240502 01:55:44.579216 28563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:55:44.602509 28563 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:55:44.603495 28563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:55:44.603863 28563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:55:44.604967 28563 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:55:44.606750 28563 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:55:44.606850 28563 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:55:44.607009 28563 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:55:44.607071 28563 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:55:44.642454 28563 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.197:32773
I20240502 01:55:44.642498 28691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.197:32773 every 8 connection(s)
I20240502 01:55:44.644006 28563 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 01:55:44.652070  9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu as pid 28563
I20240502 01:55:44.652801  9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
/tmp/dist-test-taskJZxTxO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.196:38773
--local_ip_for_outbound_sockets=127.9.176.196
--tserver_master_addrs=127.9.176.254:41377
--webserver_port=37483
--webserver_interface=127.9.176.196
--builtin_ntp_servers=127.9.176.212:46301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:55:44.658272 28692 heartbeater.cc:347] Connected to a master server at 127.9.176.254:41377
I20240502 01:55:44.658689 28692 heartbeater.cc:464] Registering TS with master...
I20240502 01:55:44.659919 28692 heartbeater.cc:511] Master 127.9.176.254:41377 requested a full tablet report, sending...
I20240502 01:55:44.662232 27566 ts_manager.cc:194] Re-registered known tserver with Master: c541378ca18944c49c37f49097bc7f81 (127.9.176.197:32773)
I20240502 01:55:44.663594 27566 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.176.197:35131
W20240502 01:55:45.031169 28696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:55:45.031481 28696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:55:45.031625 28696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:55:45.039528 28696 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:55:45.039690 28696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:55:45.039855 28696 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.196
I20240502 01:55:45.050215 28696 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:46301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.196:38773
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.9.176.196
--webserver_port=37483
--tserver_master_addrs=127.9.176.254:41377
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.196
--log_dir=/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:55:45.051589 28696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:55:45.053491 28696 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:55:45.066866 28701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:45.067301 28704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:45.068068 28702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:55:45.072741 28696 server_base.cc:1034] running on GCE node
I20240502 01:55:45.073498 28696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:55:45.074626 28696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:55:45.075862 28696 hybrid_clock.cc:648] HybridClock initialized: now 1714614945075804 us; error 57 us; skew 500 ppm
I20240502 01:55:45.076270 28696 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:55:45.078941 28696 webserver.cc:460] Webserver started at http://127.9.176.196:37483/ using document root <none> and password file <none>
I20240502 01:55:45.079890 28696 fs_manager.cc:359] Metadata directory not provided
I20240502 01:55:45.080049 28696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:55:45.086567 28696 fs_manager.cc:709] Time spent opening directory manager: real 0.004s	user 0.003s	sys 0.002s
I20240502 01:55:45.089918 28710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:55:45.091727 28696 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.000s
I20240502 01:55:45.091954 28696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "543169fd701141c181c55566d2cae00d"
format_stamp: "Formatted at 2024-05-02 01:55:35 on dist-test-slave-j3hh"
I20240502 01:55:45.092960 28696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:55:45.115475 28696 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:55:45.116385 28696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:55:45.116672 28696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:55:45.117708 28696 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:55:45.119375 28696 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:55:45.119460 28696 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:55:45.119598 28696 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:55:45.119668 28696 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.001s	sys 0.000s
I20240502 01:55:45.152920 28696 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.196:38773
I20240502 01:55:45.152966 28824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.196:38773 every 8 connection(s)
I20240502 01:55:45.154659 28696 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 01:55:45.163153  9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu as pid 28696
I20240502 01:55:45.169117 28825 heartbeater.cc:347] Connected to a master server at 127.9.176.254:41377
I20240502 01:55:45.169494 28825 heartbeater.cc:464] Registering TS with master...
I20240502 01:55:45.170457 28825 heartbeater.cc:511] Master 127.9.176.254:41377 requested a full tablet report, sending...
I20240502 01:55:45.172271 27566 ts_manager.cc:194] Re-registered known tserver with Master: 543169fd701141c181c55566d2cae00d (127.9.176.196:38773)
I20240502 01:55:45.173101 27566 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.176.196:40875
I20240502 01:55:45.667225 28692 heartbeater.cc:503] Master 127.9.176.254:41377 was elected leader, sending a full tablet report...
I20240502 01:55:45.690212 28758 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:45.709977 27827 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:55:45.745203 27696 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:55:45.753904 28626 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:45.755497 27962 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:45.929906 28758 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:45.938999 27827 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:55:45.942221 27962 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:45.950155 27696 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:55:45.951287 28626 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:46.142665 28758 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:46.150696 27962 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:46.155496 28626 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:46.176710 28825 heartbeater.cc:503] Master 127.9.176.254:41377 was elected leader, sending a full tablet report...
I20240502 01:55:46.183558 27696 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:55:46.183719 27827 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:55:46.344266 28758 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:46.346740 27962 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:46.362493 27696 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:55:46.370270 28626 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:46.384778 27827 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:55:46.462309 27715 consensus_queue.cc:237] T 9a16d465aa2b426484868a2b212c5d00 P d50f07af87d74f039fb2b43cf166fa43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } attrs { replace: true } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: true } }
I20240502 01:55:46.465633 27848 raft_consensus.cc:1261] T 9a16d465aa2b426484868a2b212c5d00 P a64152fd9595478ca767a9f248ef3b85 [term 1 FOLLOWER]: Refusing update from remote peer d50f07af87d74f039fb2b43cf166fa43: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:46.466681 27983 raft_consensus.cc:1261] T 9a16d465aa2b426484868a2b212c5d00 P b550bed5a64f466c8ba5a1761f3bef39 [term 1 FOLLOWER]: Refusing update from remote peer d50f07af87d74f039fb2b43cf166fa43: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:46.467170 28973 consensus_queue.cc:1035] T 9a16d465aa2b426484868a2b212c5d00 P d50f07af87d74f039fb2b43cf166fa43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:46.467650 28479 consensus_queue.cc:1035] T 9a16d465aa2b426484868a2b212c5d00 P d50f07af87d74f039fb2b43cf166fa43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:46.471941 28973 raft_consensus.cc:2932] T 9a16d465aa2b426484868a2b212c5d00 P d50f07af87d74f039fb2b43cf166fa43 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 543169fd701141c181c55566d2cae00d (127.9.176.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } attrs { replace: true } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: true } } }
I20240502 01:55:46.472826 27848 raft_consensus.cc:2932] T 9a16d465aa2b426484868a2b212c5d00 P a64152fd9595478ca767a9f248ef3b85 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 543169fd701141c181c55566d2cae00d (127.9.176.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } attrs { replace: true } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: true } } }
W20240502 01:55:46.476024 27649 consensus_peers.cc:485] T 9a16d465aa2b426484868a2b212c5d00 P d50f07af87d74f039fb2b43cf166fa43 -> Peer 543169fd701141c181c55566d2cae00d (127.9.176.196:38773): Couldn't send request to peer 543169fd701141c181c55566d2cae00d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9a16d465aa2b426484868a2b212c5d00. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:46.476843 27983 raft_consensus.cc:2932] T 9a16d465aa2b426484868a2b212c5d00 P b550bed5a64f466c8ba5a1761f3bef39 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 543169fd701141c181c55566d2cae00d (127.9.176.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } attrs { replace: true } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: true } } }
I20240502 01:55:46.481886 27569 catalog_manager.cc:5484] T 9a16d465aa2b426484868a2b212c5d00 P d50f07af87d74f039fb2b43cf166fa43 reported cstate change: config changed from index -1 to 4, NON_VOTER 543169fd701141c181c55566d2cae00d (127.9.176.196) added. New cstate: current_term: 1 leader_uuid: "d50f07af87d74f039fb2b43cf166fa43" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:55:46.497057 27848 consensus_queue.cc:237] T 5e8b27dca9af473188c50269820343f1 P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } attrs { replace: true } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: true } }
I20240502 01:55:46.500312 27715 raft_consensus.cc:1261] T 5e8b27dca9af473188c50269820343f1 P d50f07af87d74f039fb2b43cf166fa43 [term 1 FOLLOWER]: Refusing update from remote peer a64152fd9595478ca767a9f248ef3b85: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:46.501104 28956 consensus_queue.cc:1035] T 5e8b27dca9af473188c50269820343f1 P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:46.504036 27983 raft_consensus.cc:1261] T 5e8b27dca9af473188c50269820343f1 P b550bed5a64f466c8ba5a1761f3bef39 [term 1 FOLLOWER]: Refusing update from remote peer a64152fd9595478ca767a9f248ef3b85: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:46.504822 28866 consensus_queue.cc:1035] T 5e8b27dca9af473188c50269820343f1 P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:46.505342 28991 raft_consensus.cc:2932] T 5e8b27dca9af473188c50269820343f1 P a64152fd9595478ca767a9f248ef3b85 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c541378ca18944c49c37f49097bc7f81 (127.9.176.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } attrs { replace: true } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: true } } }
I20240502 01:55:46.506541 27715 raft_consensus.cc:2932] T 5e8b27dca9af473188c50269820343f1 P d50f07af87d74f039fb2b43cf166fa43 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c541378ca18944c49c37f49097bc7f81 (127.9.176.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } attrs { replace: true } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: true } } }
W20240502 01:55:46.510496 27784 consensus_peers.cc:485] T 5e8b27dca9af473188c50269820343f1 P a64152fd9595478ca767a9f248ef3b85 -> Peer c541378ca18944c49c37f49097bc7f81 (127.9.176.197:32773): Couldn't send request to peer c541378ca18944c49c37f49097bc7f81. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5e8b27dca9af473188c50269820343f1. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:46.511929 27566 catalog_manager.cc:5484] T 5e8b27dca9af473188c50269820343f1 P a64152fd9595478ca767a9f248ef3b85 reported cstate change: config changed from index -1 to 4, NON_VOTER c541378ca18944c49c37f49097bc7f81 (127.9.176.197) added. New cstate: current_term: 1 leader_uuid: "a64152fd9595478ca767a9f248ef3b85" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:55:46.517921 27983 raft_consensus.cc:2932] T 5e8b27dca9af473188c50269820343f1 P b550bed5a64f466c8ba5a1761f3bef39 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c541378ca18944c49c37f49097bc7f81 (127.9.176.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } attrs { replace: true } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: true } } }
I20240502 01:55:46.534552 27848 consensus_queue.cc:237] T d404c85b11594e7eb1257fcf384beafa P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } attrs { replace: true } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: true } }
I20240502 01:55:46.537216 27715 raft_consensus.cc:1261] T d404c85b11594e7eb1257fcf384beafa P d50f07af87d74f039fb2b43cf166fa43 [term 1 FOLLOWER]: Refusing update from remote peer a64152fd9595478ca767a9f248ef3b85: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:46.537386 27983 raft_consensus.cc:1261] T d404c85b11594e7eb1257fcf384beafa P b550bed5a64f466c8ba5a1761f3bef39 [term 1 FOLLOWER]: Refusing update from remote peer a64152fd9595478ca767a9f248ef3b85: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:46.537930 28956 consensus_queue.cc:1035] T d404c85b11594e7eb1257fcf384beafa P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:46.538291 28866 consensus_queue.cc:1035] T d404c85b11594e7eb1257fcf384beafa P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:46.541383 28956 raft_consensus.cc:2932] T d404c85b11594e7eb1257fcf384beafa P a64152fd9595478ca767a9f248ef3b85 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 543169fd701141c181c55566d2cae00d (127.9.176.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } attrs { replace: true } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: true } } }
I20240502 01:55:46.542706 27983 raft_consensus.cc:2932] T d404c85b11594e7eb1257fcf384beafa P b550bed5a64f466c8ba5a1761f3bef39 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 543169fd701141c181c55566d2cae00d (127.9.176.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } attrs { replace: true } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: true } } }
W20240502 01:55:46.544519 27782 consensus_peers.cc:485] T d404c85b11594e7eb1257fcf384beafa P a64152fd9595478ca767a9f248ef3b85 -> Peer 543169fd701141c181c55566d2cae00d (127.9.176.196:38773): Couldn't send request to peer 543169fd701141c181c55566d2cae00d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d404c85b11594e7eb1257fcf384beafa. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:46.544368 27715 raft_consensus.cc:2932] T d404c85b11594e7eb1257fcf384beafa P d50f07af87d74f039fb2b43cf166fa43 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 543169fd701141c181c55566d2cae00d (127.9.176.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } attrs { replace: true } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: true } } }
I20240502 01:55:46.547123 27566 catalog_manager.cc:5484] T d404c85b11594e7eb1257fcf384beafa P a64152fd9595478ca767a9f248ef3b85 reported cstate change: config changed from index -1 to 4, NON_VOTER 543169fd701141c181c55566d2cae00d (127.9.176.196) added. New cstate: current_term: 1 leader_uuid: "a64152fd9595478ca767a9f248ef3b85" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:55:46.563277 27848 consensus_queue.cc:237] T 31de6c641c094cce894d8513b878350c P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: true } }
I20240502 01:55:46.565779 27983 raft_consensus.cc:1261] T 31de6c641c094cce894d8513b878350c P b550bed5a64f466c8ba5a1761f3bef39 [term 1 FOLLOWER]: Refusing update from remote peer a64152fd9595478ca767a9f248ef3b85: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:46.565932 27715 raft_consensus.cc:1261] T 31de6c641c094cce894d8513b878350c P d50f07af87d74f039fb2b43cf166fa43 [term 1 FOLLOWER]: Refusing update from remote peer a64152fd9595478ca767a9f248ef3b85: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 01:55:46.566005 27784 consensus_peers.cc:485] T 31de6c641c094cce894d8513b878350c P a64152fd9595478ca767a9f248ef3b85 -> Peer c541378ca18944c49c37f49097bc7f81 (127.9.176.197:32773): Couldn't send request to peer c541378ca18944c49c37f49097bc7f81. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 31de6c641c094cce894d8513b878350c. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:46.566498 28956 consensus_queue.cc:1035] T 31de6c641c094cce894d8513b878350c P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:46.566860 28991 consensus_queue.cc:1035] T 31de6c641c094cce894d8513b878350c P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:46.570611 28866 raft_consensus.cc:2932] T 31de6c641c094cce894d8513b878350c P a64152fd9595478ca767a9f248ef3b85 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c541378ca18944c49c37f49097bc7f81 (127.9.176.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: true } } }
I20240502 01:55:46.571435 27983 raft_consensus.cc:2932] T 31de6c641c094cce894d8513b878350c P b550bed5a64f466c8ba5a1761f3bef39 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c541378ca18944c49c37f49097bc7f81 (127.9.176.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: true } } }
I20240502 01:55:46.571430 27715 raft_consensus.cc:2932] T 31de6c641c094cce894d8513b878350c P d50f07af87d74f039fb2b43cf166fa43 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c541378ca18944c49c37f49097bc7f81 (127.9.176.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: true } } }
I20240502 01:55:46.576624 27570 catalog_manager.cc:5484] T 31de6c641c094cce894d8513b878350c P b550bed5a64f466c8ba5a1761f3bef39 reported cstate change: config changed from index -1 to 4, NON_VOTER c541378ca18944c49c37f49097bc7f81 (127.9.176.197) added. New cstate: current_term: 1 leader_uuid: "a64152fd9595478ca767a9f248ef3b85" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: true } } }
I20240502 01:55:46.592310 27715 consensus_queue.cc:237] T f22c0dfa05664a4a9f10aa81424a396e P d50f07af87d74f039fb2b43cf166fa43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } attrs { replace: true } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: true } }
I20240502 01:55:46.595073 27983 raft_consensus.cc:1261] T f22c0dfa05664a4a9f10aa81424a396e P b550bed5a64f466c8ba5a1761f3bef39 [term 1 FOLLOWER]: Refusing update from remote peer d50f07af87d74f039fb2b43cf166fa43: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:46.595075 27848 raft_consensus.cc:1261] T f22c0dfa05664a4a9f10aa81424a396e P a64152fd9595478ca767a9f248ef3b85 [term 1 FOLLOWER]: Refusing update from remote peer d50f07af87d74f039fb2b43cf166fa43: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 01:55:46.595338 27649 consensus_peers.cc:485] T f22c0dfa05664a4a9f10aa81424a396e P d50f07af87d74f039fb2b43cf166fa43 -> Peer 543169fd701141c181c55566d2cae00d (127.9.176.196:38773): Couldn't send request to peer 543169fd701141c181c55566d2cae00d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f22c0dfa05664a4a9f10aa81424a396e. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:46.595741 28977 consensus_queue.cc:1035] T f22c0dfa05664a4a9f10aa81424a396e P d50f07af87d74f039fb2b43cf166fa43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:46.596084 28973 consensus_queue.cc:1035] T f22c0dfa05664a4a9f10aa81424a396e P d50f07af87d74f039fb2b43cf166fa43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:46.599601 28977 raft_consensus.cc:2932] T f22c0dfa05664a4a9f10aa81424a396e P d50f07af87d74f039fb2b43cf166fa43 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 543169fd701141c181c55566d2cae00d (127.9.176.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } attrs { replace: true } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: true } } }
I20240502 01:55:46.600571 27983 raft_consensus.cc:2932] T f22c0dfa05664a4a9f10aa81424a396e P b550bed5a64f466c8ba5a1761f3bef39 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 543169fd701141c181c55566d2cae00d (127.9.176.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } attrs { replace: true } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: true } } }
I20240502 01:55:46.601014 27848 raft_consensus.cc:2932] T f22c0dfa05664a4a9f10aa81424a396e P a64152fd9595478ca767a9f248ef3b85 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 543169fd701141c181c55566d2cae00d (127.9.176.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } attrs { replace: true } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: true } } }
I20240502 01:55:46.606777 27569 catalog_manager.cc:5484] T f22c0dfa05664a4a9f10aa81424a396e P d50f07af87d74f039fb2b43cf166fa43 reported cstate change: config changed from index -1 to 4, NON_VOTER 543169fd701141c181c55566d2cae00d (127.9.176.196) added. New cstate: current_term: 1 leader_uuid: "d50f07af87d74f039fb2b43cf166fa43" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:55:46.619805 27848 consensus_queue.cc:237] T f1597c56a1a74b37bb5e18e8c4662f46 P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: true } }
I20240502 01:55:46.622169 27715 raft_consensus.cc:1261] T f1597c56a1a74b37bb5e18e8c4662f46 P d50f07af87d74f039fb2b43cf166fa43 [term 1 FOLLOWER]: Refusing update from remote peer a64152fd9595478ca767a9f248ef3b85: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:46.622433 27983 raft_consensus.cc:1261] T f1597c56a1a74b37bb5e18e8c4662f46 P b550bed5a64f466c8ba5a1761f3bef39 [term 1 FOLLOWER]: Refusing update from remote peer a64152fd9595478ca767a9f248ef3b85: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 01:55:46.622498 27784 consensus_peers.cc:485] T f1597c56a1a74b37bb5e18e8c4662f46 P a64152fd9595478ca767a9f248ef3b85 -> Peer c541378ca18944c49c37f49097bc7f81 (127.9.176.197:32773): Couldn't send request to peer c541378ca18944c49c37f49097bc7f81. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f1597c56a1a74b37bb5e18e8c4662f46. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:46.622845 28866 consensus_queue.cc:1035] T f1597c56a1a74b37bb5e18e8c4662f46 P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:46.623178 28956 consensus_queue.cc:1035] T f1597c56a1a74b37bb5e18e8c4662f46 P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:46.626116 28866 raft_consensus.cc:2932] T f1597c56a1a74b37bb5e18e8c4662f46 P a64152fd9595478ca767a9f248ef3b85 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c541378ca18944c49c37f49097bc7f81 (127.9.176.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: true } } }
I20240502 01:55:46.626797 27983 raft_consensus.cc:2932] T f1597c56a1a74b37bb5e18e8c4662f46 P b550bed5a64f466c8ba5a1761f3bef39 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c541378ca18944c49c37f49097bc7f81 (127.9.176.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: true } } }
I20240502 01:55:46.627431 27715 raft_consensus.cc:2932] T f1597c56a1a74b37bb5e18e8c4662f46 P d50f07af87d74f039fb2b43cf166fa43 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c541378ca18944c49c37f49097bc7f81 (127.9.176.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: true } } }
I20240502 01:55:46.631429 27566 catalog_manager.cc:5484] T f1597c56a1a74b37bb5e18e8c4662f46 P b550bed5a64f466c8ba5a1761f3bef39 reported cstate change: config changed from index -1 to 4, NON_VOTER c541378ca18944c49c37f49097bc7f81 (127.9.176.197) added. New cstate: current_term: 1 leader_uuid: "a64152fd9595478ca767a9f248ef3b85" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: true } } }
I20240502 01:55:46.946311 29083 ts_tablet_manager.cc:902] T 5e8b27dca9af473188c50269820343f1 P c541378ca18944c49c37f49097bc7f81: Initiating tablet copy from peer a64152fd9595478ca767a9f248ef3b85 (127.9.176.194:38689)
I20240502 01:55:46.950476 29083 tablet_copy_client.cc:308] T 5e8b27dca9af473188c50269820343f1 P c541378ca18944c49c37f49097bc7f81: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.194:38689
I20240502 01:55:46.954814 29093 ts_tablet_manager.cc:902] T 9a16d465aa2b426484868a2b212c5d00 P 543169fd701141c181c55566d2cae00d: Initiating tablet copy from peer d50f07af87d74f039fb2b43cf166fa43 (127.9.176.193:34203)
I20240502 01:55:46.970151 29093 tablet_copy_client.cc:308] T 9a16d465aa2b426484868a2b212c5d00 P 543169fd701141c181c55566d2cae00d: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.193:34203
I20240502 01:55:46.974056 27869 tablet_copy_service.cc:140] P a64152fd9595478ca767a9f248ef3b85: Received BeginTabletCopySession request for tablet 5e8b27dca9af473188c50269820343f1 from peer c541378ca18944c49c37f49097bc7f81 ({username='slave'} at 127.9.176.197:47985)
I20240502 01:55:46.974400 27869 tablet_copy_service.cc:161] P a64152fd9595478ca767a9f248ef3b85: Beginning new tablet copy session on tablet 5e8b27dca9af473188c50269820343f1 from peer c541378ca18944c49c37f49097bc7f81 at {username='slave'} at 127.9.176.197:47985: session id = c541378ca18944c49c37f49097bc7f81-5e8b27dca9af473188c50269820343f1
I20240502 01:55:46.979012 27869 tablet_copy_source_session.cc:205] T 5e8b27dca9af473188c50269820343f1 P a64152fd9595478ca767a9f248ef3b85: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:46.979168 27737 tablet_copy_service.cc:140] P d50f07af87d74f039fb2b43cf166fa43: Received BeginTabletCopySession request for tablet 9a16d465aa2b426484868a2b212c5d00 from peer 543169fd701141c181c55566d2cae00d ({username='slave'} at 127.9.176.196:53953)
I20240502 01:55:46.979465 27737 tablet_copy_service.cc:161] P d50f07af87d74f039fb2b43cf166fa43: Beginning new tablet copy session on tablet 9a16d465aa2b426484868a2b212c5d00 from peer 543169fd701141c181c55566d2cae00d at {username='slave'} at 127.9.176.196:53953: session id = 543169fd701141c181c55566d2cae00d-9a16d465aa2b426484868a2b212c5d00
I20240502 01:55:46.982602 27737 tablet_copy_source_session.cc:205] T 9a16d465aa2b426484868a2b212c5d00 P d50f07af87d74f039fb2b43cf166fa43: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:46.984416 29083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e8b27dca9af473188c50269820343f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:46.987630 29093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a16d465aa2b426484868a2b212c5d00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:46.996482 29083 tablet_copy_client.cc:783] T 5e8b27dca9af473188c50269820343f1 P c541378ca18944c49c37f49097bc7f81: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:46.996853 29083 tablet_copy_client.cc:647] T 5e8b27dca9af473188c50269820343f1 P c541378ca18944c49c37f49097bc7f81: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:46.999672 29083 tablet_copy_client.cc:523] T 5e8b27dca9af473188c50269820343f1 P c541378ca18944c49c37f49097bc7f81: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:46.999665 29093 tablet_copy_client.cc:783] T 9a16d465aa2b426484868a2b212c5d00 P 543169fd701141c181c55566d2cae00d: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:47.000015 29093 tablet_copy_client.cc:647] T 9a16d465aa2b426484868a2b212c5d00 P 543169fd701141c181c55566d2cae00d: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:47.003479 29093 tablet_copy_client.cc:523] T 9a16d465aa2b426484868a2b212c5d00 P 543169fd701141c181c55566d2cae00d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:47.004809 29083 tablet_bootstrap.cc:492] T 5e8b27dca9af473188c50269820343f1 P c541378ca18944c49c37f49097bc7f81: Bootstrap starting.
I20240502 01:55:47.008359 29093 tablet_bootstrap.cc:492] T 9a16d465aa2b426484868a2b212c5d00 P 543169fd701141c181c55566d2cae00d: Bootstrap starting.
I20240502 01:55:47.013700 29102 ts_tablet_manager.cc:902] T d404c85b11594e7eb1257fcf384beafa P 543169fd701141c181c55566d2cae00d: Initiating tablet copy from peer a64152fd9595478ca767a9f248ef3b85 (127.9.176.194:38689)
I20240502 01:55:47.015851 29102 tablet_copy_client.cc:308] T d404c85b11594e7eb1257fcf384beafa P 543169fd701141c181c55566d2cae00d: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.194:38689
I20240502 01:55:47.019979 29083 log.cc:824] T 5e8b27dca9af473188c50269820343f1 P c541378ca18944c49c37f49097bc7f81: Log is configured to *not* fsync() on all Append() calls
I20240502 01:55:47.022712 27869 tablet_copy_service.cc:140] P a64152fd9595478ca767a9f248ef3b85: Received BeginTabletCopySession request for tablet d404c85b11594e7eb1257fcf384beafa from peer 543169fd701141c181c55566d2cae00d ({username='slave'} at 127.9.176.196:38793)
I20240502 01:55:47.022909 27869 tablet_copy_service.cc:161] P a64152fd9595478ca767a9f248ef3b85: Beginning new tablet copy session on tablet d404c85b11594e7eb1257fcf384beafa from peer 543169fd701141c181c55566d2cae00d at {username='slave'} at 127.9.176.196:38793: session id = 543169fd701141c181c55566d2cae00d-d404c85b11594e7eb1257fcf384beafa
I20240502 01:55:47.023291 29093 log.cc:824] T 9a16d465aa2b426484868a2b212c5d00 P 543169fd701141c181c55566d2cae00d: Log is configured to *not* fsync() on all Append() calls
I20240502 01:55:47.025739 27869 tablet_copy_source_session.cc:205] T d404c85b11594e7eb1257fcf384beafa P a64152fd9595478ca767a9f248ef3b85: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:47.027295 29102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d404c85b11594e7eb1257fcf384beafa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:47.039183 29102 tablet_copy_client.cc:783] T d404c85b11594e7eb1257fcf384beafa P 543169fd701141c181c55566d2cae00d: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:47.039608 29102 tablet_copy_client.cc:647] T d404c85b11594e7eb1257fcf384beafa P 543169fd701141c181c55566d2cae00d: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:47.041903 29102 tablet_copy_client.cc:523] T d404c85b11594e7eb1257fcf384beafa P 543169fd701141c181c55566d2cae00d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:47.042501 29109 ts_tablet_manager.cc:902] T f22c0dfa05664a4a9f10aa81424a396e P 543169fd701141c181c55566d2cae00d: Initiating tablet copy from peer d50f07af87d74f039fb2b43cf166fa43 (127.9.176.193:34203)
I20240502 01:55:47.046650 29102 tablet_bootstrap.cc:492] T d404c85b11594e7eb1257fcf384beafa P 543169fd701141c181c55566d2cae00d: Bootstrap starting.
I20240502 01:55:47.071833 29083 tablet_bootstrap.cc:492] T 5e8b27dca9af473188c50269820343f1 P c541378ca18944c49c37f49097bc7f81: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:47.072553 29083 tablet_bootstrap.cc:492] T 5e8b27dca9af473188c50269820343f1 P c541378ca18944c49c37f49097bc7f81: Bootstrap complete.
I20240502 01:55:47.073140 29083 ts_tablet_manager.cc:1367] T 5e8b27dca9af473188c50269820343f1 P c541378ca18944c49c37f49097bc7f81: Time spent bootstrapping tablet: real 0.069s	user 0.062s	sys 0.004s
I20240502 01:55:47.074286 29109 tablet_copy_client.cc:308] T f22c0dfa05664a4a9f10aa81424a396e P 543169fd701141c181c55566d2cae00d: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.193:34203
I20240502 01:55:47.075400 27737 tablet_copy_service.cc:140] P d50f07af87d74f039fb2b43cf166fa43: Received BeginTabletCopySession request for tablet f22c0dfa05664a4a9f10aa81424a396e from peer 543169fd701141c181c55566d2cae00d ({username='slave'} at 127.9.176.196:53953)
I20240502 01:55:47.075650 27737 tablet_copy_service.cc:161] P d50f07af87d74f039fb2b43cf166fa43: Beginning new tablet copy session on tablet f22c0dfa05664a4a9f10aa81424a396e from peer 543169fd701141c181c55566d2cae00d at {username='slave'} at 127.9.176.196:53953: session id = 543169fd701141c181c55566d2cae00d-f22c0dfa05664a4a9f10aa81424a396e
I20240502 01:55:47.078567 29093 tablet_bootstrap.cc:492] T 9a16d465aa2b426484868a2b212c5d00 P 543169fd701141c181c55566d2cae00d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:47.078847 27737 tablet_copy_source_session.cc:205] T f22c0dfa05664a4a9f10aa81424a396e P d50f07af87d74f039fb2b43cf166fa43: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:47.079277 29093 tablet_bootstrap.cc:492] T 9a16d465aa2b426484868a2b212c5d00 P 543169fd701141c181c55566d2cae00d: Bootstrap complete.
I20240502 01:55:47.079910 29093 ts_tablet_manager.cc:1367] T 9a16d465aa2b426484868a2b212c5d00 P 543169fd701141c181c55566d2cae00d: Time spent bootstrapping tablet: real 0.072s	user 0.060s	sys 0.008s
I20240502 01:55:47.080348 29083 raft_consensus.cc:348] T 5e8b27dca9af473188c50269820343f1 P c541378ca18944c49c37f49097bc7f81 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } attrs { replace: true } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: true } }
I20240502 01:55:47.080932 29083 raft_consensus.cc:729] T 5e8b27dca9af473188c50269820343f1 P c541378ca18944c49c37f49097bc7f81 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c541378ca18944c49c37f49097bc7f81, State: Initialized, Role: LEARNER
I20240502 01:55:47.081080 29109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f22c0dfa05664a4a9f10aa81424a396e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:47.081574 29083 consensus_queue.cc:260] T 5e8b27dca9af473188c50269820343f1 P c541378ca18944c49c37f49097bc7f81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } attrs { replace: true } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: true } }
I20240502 01:55:47.089671 29109 tablet_copy_client.cc:783] T f22c0dfa05664a4a9f10aa81424a396e P 543169fd701141c181c55566d2cae00d: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:47.090106 29109 tablet_copy_client.cc:647] T f22c0dfa05664a4a9f10aa81424a396e P 543169fd701141c181c55566d2cae00d: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:47.090212 29083 ts_tablet_manager.cc:1398] T 5e8b27dca9af473188c50269820343f1 P c541378ca18944c49c37f49097bc7f81: Time spent starting tablet: real 0.017s	user 0.013s	sys 0.004s
I20240502 01:55:47.092981 27869 tablet_copy_service.cc:342] P a64152fd9595478ca767a9f248ef3b85: Request end of tablet copy session c541378ca18944c49c37f49097bc7f81-5e8b27dca9af473188c50269820343f1 received from {username='slave'} at 127.9.176.197:47985
I20240502 01:55:47.093199 27869 tablet_copy_service.cc:433] P a64152fd9595478ca767a9f248ef3b85: ending tablet copy session c541378ca18944c49c37f49097bc7f81-5e8b27dca9af473188c50269820343f1 on tablet 5e8b27dca9af473188c50269820343f1 with peer c541378ca18944c49c37f49097bc7f81
I20240502 01:55:47.093070 29093 raft_consensus.cc:348] T 9a16d465aa2b426484868a2b212c5d00 P 543169fd701141c181c55566d2cae00d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } attrs { replace: true } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: true } }
I20240502 01:55:47.093510 29093 raft_consensus.cc:729] T 9a16d465aa2b426484868a2b212c5d00 P 543169fd701141c181c55566d2cae00d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 543169fd701141c181c55566d2cae00d, State: Initialized, Role: LEARNER
I20240502 01:55:47.094175 29109 tablet_copy_client.cc:523] T f22c0dfa05664a4a9f10aa81424a396e P 543169fd701141c181c55566d2cae00d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:47.094213 29093 consensus_queue.cc:260] T 9a16d465aa2b426484868a2b212c5d00 P 543169fd701141c181c55566d2cae00d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } attrs { replace: true } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: true } }
I20240502 01:55:47.098547 29109 tablet_bootstrap.cc:492] T f22c0dfa05664a4a9f10aa81424a396e P 543169fd701141c181c55566d2cae00d: Bootstrap starting.
I20240502 01:55:47.101382 29093 ts_tablet_manager.cc:1398] T 9a16d465aa2b426484868a2b212c5d00 P 543169fd701141c181c55566d2cae00d: Time spent starting tablet: real 0.021s	user 0.008s	sys 0.008s
I20240502 01:55:47.102613 27737 tablet_copy_service.cc:342] P d50f07af87d74f039fb2b43cf166fa43: Request end of tablet copy session 543169fd701141c181c55566d2cae00d-9a16d465aa2b426484868a2b212c5d00 received from {username='slave'} at 127.9.176.196:53953
I20240502 01:55:47.102806 27737 tablet_copy_service.cc:433] P d50f07af87d74f039fb2b43cf166fa43: ending tablet copy session 543169fd701141c181c55566d2cae00d-9a16d465aa2b426484868a2b212c5d00 on tablet 9a16d465aa2b426484868a2b212c5d00 with peer 543169fd701141c181c55566d2cae00d
I20240502 01:55:47.113207 29102 tablet_bootstrap.cc:492] T d404c85b11594e7eb1257fcf384beafa P 543169fd701141c181c55566d2cae00d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:47.113751 29102 tablet_bootstrap.cc:492] T d404c85b11594e7eb1257fcf384beafa P 543169fd701141c181c55566d2cae00d: Bootstrap complete.
I20240502 01:55:47.114077 29102 ts_tablet_manager.cc:1367] T d404c85b11594e7eb1257fcf384beafa P 543169fd701141c181c55566d2cae00d: Time spent bootstrapping tablet: real 0.068s	user 0.046s	sys 0.019s
I20240502 01:55:47.115197 29102 raft_consensus.cc:348] T d404c85b11594e7eb1257fcf384beafa P 543169fd701141c181c55566d2cae00d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } attrs { replace: true } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: true } }
I20240502 01:55:47.115514 29102 raft_consensus.cc:729] T d404c85b11594e7eb1257fcf384beafa P 543169fd701141c181c55566d2cae00d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 543169fd701141c181c55566d2cae00d, State: Initialized, Role: LEARNER
I20240502 01:55:47.115757 29102 consensus_queue.cc:260] T d404c85b11594e7eb1257fcf384beafa P 543169fd701141c181c55566d2cae00d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } attrs { replace: true } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: true } }
I20240502 01:55:47.116384 29102 ts_tablet_manager.cc:1398] T d404c85b11594e7eb1257fcf384beafa P 543169fd701141c181c55566d2cae00d: Time spent starting tablet: real 0.002s	user 0.004s	sys 0.000s
I20240502 01:55:47.117321 27869 tablet_copy_service.cc:342] P a64152fd9595478ca767a9f248ef3b85: Request end of tablet copy session 543169fd701141c181c55566d2cae00d-d404c85b11594e7eb1257fcf384beafa received from {username='slave'} at 127.9.176.196:38793
I20240502 01:55:47.117470 27869 tablet_copy_service.cc:433] P a64152fd9595478ca767a9f248ef3b85: ending tablet copy session 543169fd701141c181c55566d2cae00d-d404c85b11594e7eb1257fcf384beafa on tablet d404c85b11594e7eb1257fcf384beafa with peer 543169fd701141c181c55566d2cae00d
I20240502 01:55:47.128983 29083 ts_tablet_manager.cc:902] T 31de6c641c094cce894d8513b878350c P c541378ca18944c49c37f49097bc7f81: Initiating tablet copy from peer a64152fd9595478ca767a9f248ef3b85 (127.9.176.194:38689)
I20240502 01:55:47.130308 29083 tablet_copy_client.cc:308] T 31de6c641c094cce894d8513b878350c P c541378ca18944c49c37f49097bc7f81: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.194:38689
I20240502 01:55:47.131067 27869 tablet_copy_service.cc:140] P a64152fd9595478ca767a9f248ef3b85: Received BeginTabletCopySession request for tablet 31de6c641c094cce894d8513b878350c from peer c541378ca18944c49c37f49097bc7f81 ({username='slave'} at 127.9.176.197:47985)
I20240502 01:55:47.131258 27869 tablet_copy_service.cc:161] P a64152fd9595478ca767a9f248ef3b85: Beginning new tablet copy session on tablet 31de6c641c094cce894d8513b878350c from peer c541378ca18944c49c37f49097bc7f81 at {username='slave'} at 127.9.176.197:47985: session id = c541378ca18944c49c37f49097bc7f81-31de6c641c094cce894d8513b878350c
I20240502 01:55:47.134439 27869 tablet_copy_source_session.cc:205] T 31de6c641c094cce894d8513b878350c P a64152fd9595478ca767a9f248ef3b85: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:47.135749 29083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31de6c641c094cce894d8513b878350c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:47.140650 29083 tablet_copy_client.cc:783] T 31de6c641c094cce894d8513b878350c P c541378ca18944c49c37f49097bc7f81: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:47.140937 29083 tablet_copy_client.cc:647] T 31de6c641c094cce894d8513b878350c P c541378ca18944c49c37f49097bc7f81: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:47.141005 29109 tablet_bootstrap.cc:492] T f22c0dfa05664a4a9f10aa81424a396e P 543169fd701141c181c55566d2cae00d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:47.141389 29109 tablet_bootstrap.cc:492] T f22c0dfa05664a4a9f10aa81424a396e P 543169fd701141c181c55566d2cae00d: Bootstrap complete.
I20240502 01:55:47.141697 29109 ts_tablet_manager.cc:1367] T f22c0dfa05664a4a9f10aa81424a396e P 543169fd701141c181c55566d2cae00d: Time spent bootstrapping tablet: real 0.043s	user 0.037s	sys 0.003s
I20240502 01:55:47.142469 29109 raft_consensus.cc:348] T f22c0dfa05664a4a9f10aa81424a396e P 543169fd701141c181c55566d2cae00d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } attrs { replace: true } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: true } }
I20240502 01:55:47.142592 29083 tablet_copy_client.cc:523] T 31de6c641c094cce894d8513b878350c P c541378ca18944c49c37f49097bc7f81: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:47.142671 29109 raft_consensus.cc:729] T f22c0dfa05664a4a9f10aa81424a396e P 543169fd701141c181c55566d2cae00d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 543169fd701141c181c55566d2cae00d, State: Initialized, Role: LEARNER
I20240502 01:55:47.142871 29109 consensus_queue.cc:260] T f22c0dfa05664a4a9f10aa81424a396e P 543169fd701141c181c55566d2cae00d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } attrs { replace: true } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: true } }
I20240502 01:55:47.143307 29109 ts_tablet_manager.cc:1398] T f22c0dfa05664a4a9f10aa81424a396e P 543169fd701141c181c55566d2cae00d: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240502 01:55:47.144130 27737 tablet_copy_service.cc:342] P d50f07af87d74f039fb2b43cf166fa43: Request end of tablet copy session 543169fd701141c181c55566d2cae00d-f22c0dfa05664a4a9f10aa81424a396e received from {username='slave'} at 127.9.176.196:53953
I20240502 01:55:47.144274 27737 tablet_copy_service.cc:433] P d50f07af87d74f039fb2b43cf166fa43: ending tablet copy session 543169fd701141c181c55566d2cae00d-f22c0dfa05664a4a9f10aa81424a396e on tablet f22c0dfa05664a4a9f10aa81424a396e with peer 543169fd701141c181c55566d2cae00d
I20240502 01:55:47.146112 29083 tablet_bootstrap.cc:492] T 31de6c641c094cce894d8513b878350c P c541378ca18944c49c37f49097bc7f81: Bootstrap starting.
I20240502 01:55:47.180164 29083 tablet_bootstrap.cc:492] T 31de6c641c094cce894d8513b878350c P c541378ca18944c49c37f49097bc7f81: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:47.180630 29083 tablet_bootstrap.cc:492] T 31de6c641c094cce894d8513b878350c P c541378ca18944c49c37f49097bc7f81: Bootstrap complete.
I20240502 01:55:47.180989 29083 ts_tablet_manager.cc:1367] T 31de6c641c094cce894d8513b878350c P c541378ca18944c49c37f49097bc7f81: Time spent bootstrapping tablet: real 0.035s	user 0.026s	sys 0.008s
I20240502 01:55:47.182010 29083 raft_consensus.cc:348] T 31de6c641c094cce894d8513b878350c P c541378ca18944c49c37f49097bc7f81 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: true } }
I20240502 01:55:47.182319 29083 raft_consensus.cc:729] T 31de6c641c094cce894d8513b878350c P c541378ca18944c49c37f49097bc7f81 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c541378ca18944c49c37f49097bc7f81, State: Initialized, Role: LEARNER
I20240502 01:55:47.182576 29083 consensus_queue.cc:260] T 31de6c641c094cce894d8513b878350c P c541378ca18944c49c37f49097bc7f81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: true } }
I20240502 01:55:47.183151 29083 ts_tablet_manager.cc:1398] T 31de6c641c094cce894d8513b878350c P c541378ca18944c49c37f49097bc7f81: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 01:55:47.184367 27869 tablet_copy_service.cc:342] P a64152fd9595478ca767a9f248ef3b85: Request end of tablet copy session c541378ca18944c49c37f49097bc7f81-31de6c641c094cce894d8513b878350c received from {username='slave'} at 127.9.176.197:47985
I20240502 01:55:47.184566 27869 tablet_copy_service.cc:433] P a64152fd9595478ca767a9f248ef3b85: ending tablet copy session c541378ca18944c49c37f49097bc7f81-31de6c641c094cce894d8513b878350c on tablet 31de6c641c094cce894d8513b878350c with peer c541378ca18944c49c37f49097bc7f81
I20240502 01:55:47.188513 29083 ts_tablet_manager.cc:902] T f1597c56a1a74b37bb5e18e8c4662f46 P c541378ca18944c49c37f49097bc7f81: Initiating tablet copy from peer a64152fd9595478ca767a9f248ef3b85 (127.9.176.194:38689)
I20240502 01:55:47.189975 29083 tablet_copy_client.cc:308] T f1597c56a1a74b37bb5e18e8c4662f46 P c541378ca18944c49c37f49097bc7f81: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.194:38689
I20240502 01:55:47.190888 27869 tablet_copy_service.cc:140] P a64152fd9595478ca767a9f248ef3b85: Received BeginTabletCopySession request for tablet f1597c56a1a74b37bb5e18e8c4662f46 from peer c541378ca18944c49c37f49097bc7f81 ({username='slave'} at 127.9.176.197:47985)
I20240502 01:55:47.191093 27869 tablet_copy_service.cc:161] P a64152fd9595478ca767a9f248ef3b85: Beginning new tablet copy session on tablet f1597c56a1a74b37bb5e18e8c4662f46 from peer c541378ca18944c49c37f49097bc7f81 at {username='slave'} at 127.9.176.197:47985: session id = c541378ca18944c49c37f49097bc7f81-f1597c56a1a74b37bb5e18e8c4662f46
I20240502 01:55:47.193464 27869 tablet_copy_source_session.cc:205] T f1597c56a1a74b37bb5e18e8c4662f46 P a64152fd9595478ca767a9f248ef3b85: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:47.194847 29083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1597c56a1a74b37bb5e18e8c4662f46. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:47.200783 29083 tablet_copy_client.cc:783] T f1597c56a1a74b37bb5e18e8c4662f46 P c541378ca18944c49c37f49097bc7f81: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:47.201133 29083 tablet_copy_client.cc:647] T f1597c56a1a74b37bb5e18e8c4662f46 P c541378ca18944c49c37f49097bc7f81: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:47.203155 29083 tablet_copy_client.cc:523] T f1597c56a1a74b37bb5e18e8c4662f46 P c541378ca18944c49c37f49097bc7f81: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:47.206708 29083 tablet_bootstrap.cc:492] T f1597c56a1a74b37bb5e18e8c4662f46 P c541378ca18944c49c37f49097bc7f81: Bootstrap starting.
I20240502 01:55:47.253290 29083 tablet_bootstrap.cc:492] T f1597c56a1a74b37bb5e18e8c4662f46 P c541378ca18944c49c37f49097bc7f81: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:47.253858 29083 tablet_bootstrap.cc:492] T f1597c56a1a74b37bb5e18e8c4662f46 P c541378ca18944c49c37f49097bc7f81: Bootstrap complete.
I20240502 01:55:47.254207 29083 ts_tablet_manager.cc:1367] T f1597c56a1a74b37bb5e18e8c4662f46 P c541378ca18944c49c37f49097bc7f81: Time spent bootstrapping tablet: real 0.048s	user 0.043s	sys 0.002s
I20240502 01:55:47.255036 29083 raft_consensus.cc:348] T f1597c56a1a74b37bb5e18e8c4662f46 P c541378ca18944c49c37f49097bc7f81 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: true } }
I20240502 01:55:47.255300 29083 raft_consensus.cc:729] T f1597c56a1a74b37bb5e18e8c4662f46 P c541378ca18944c49c37f49097bc7f81 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c541378ca18944c49c37f49097bc7f81, State: Initialized, Role: LEARNER
I20240502 01:55:47.255635 29083 consensus_queue.cc:260] T f1597c56a1a74b37bb5e18e8c4662f46 P c541378ca18944c49c37f49097bc7f81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: true } }
I20240502 01:55:47.256258 29083 ts_tablet_manager.cc:1398] T f1597c56a1a74b37bb5e18e8c4662f46 P c541378ca18944c49c37f49097bc7f81: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 01:55:47.257752 27869 tablet_copy_service.cc:342] P a64152fd9595478ca767a9f248ef3b85: Request end of tablet copy session c541378ca18944c49c37f49097bc7f81-f1597c56a1a74b37bb5e18e8c4662f46 received from {username='slave'} at 127.9.176.197:47985
I20240502 01:55:47.257923 27869 tablet_copy_service.cc:433] P a64152fd9595478ca767a9f248ef3b85: ending tablet copy session c541378ca18944c49c37f49097bc7f81-f1597c56a1a74b37bb5e18e8c4662f46 on tablet f1597c56a1a74b37bb5e18e8c4662f46 with peer c541378ca18944c49c37f49097bc7f81
I20240502 01:55:47.342941 28646 raft_consensus.cc:1203] T 5e8b27dca9af473188c50269820343f1 P c541378ca18944c49c37f49097bc7f81 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 01:55:47.487931 28778 raft_consensus.cc:1203] T 9a16d465aa2b426484868a2b212c5d00 P 543169fd701141c181c55566d2cae00d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 01:55:47.501222 28778 raft_consensus.cc:1203] T d404c85b11594e7eb1257fcf384beafa P 543169fd701141c181c55566d2cae00d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 01:55:47.513860 28646 raft_consensus.cc:1203] T 31de6c641c094cce894d8513b878350c P c541378ca18944c49c37f49097bc7f81 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 01:55:47.645601 28778 raft_consensus.cc:1203] T f22c0dfa05664a4a9f10aa81424a396e P 543169fd701141c181c55566d2cae00d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 01:55:47.674847 28646 raft_consensus.cc:1203] T f1597c56a1a74b37bb5e18e8c4662f46 P c541378ca18944c49c37f49097bc7f81 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 01:55:47.929826 29026 raft_consensus.cc:1050] T d404c85b11594e7eb1257fcf384beafa P a64152fd9595478ca767a9f248ef3b85: attempting to promote NON_VOTER 543169fd701141c181c55566d2cae00d to VOTER
I20240502 01:55:47.930686 29026 consensus_queue.cc:237] T d404c85b11594e7eb1257fcf384beafa P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } attrs { replace: true } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } }
I20240502 01:55:47.933244 27715 raft_consensus.cc:1261] T d404c85b11594e7eb1257fcf384beafa P d50f07af87d74f039fb2b43cf166fa43 [term 1 FOLLOWER]: Refusing update from remote peer a64152fd9595478ca767a9f248ef3b85: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:47.933372 28778 raft_consensus.cc:1261] T d404c85b11594e7eb1257fcf384beafa P 543169fd701141c181c55566d2cae00d [term 1 LEARNER]: Refusing update from remote peer a64152fd9595478ca767a9f248ef3b85: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:47.933532 27983 raft_consensus.cc:1261] T d404c85b11594e7eb1257fcf384beafa P b550bed5a64f466c8ba5a1761f3bef39 [term 1 FOLLOWER]: Refusing update from remote peer a64152fd9595478ca767a9f248ef3b85: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:47.934132 29221 consensus_queue.cc:1035] T d404c85b11594e7eb1257fcf384beafa P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:47.934495 29026 consensus_queue.cc:1035] T d404c85b11594e7eb1257fcf384beafa P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:47.934808 28866 consensus_queue.cc:1035] T d404c85b11594e7eb1257fcf384beafa P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:47.938866 28956 raft_consensus.cc:2932] T d404c85b11594e7eb1257fcf384beafa P a64152fd9595478ca767a9f248ef3b85 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 543169fd701141c181c55566d2cae00d (127.9.176.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } attrs { replace: true } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } }
I20240502 01:55:47.939530 27983 raft_consensus.cc:2932] T d404c85b11594e7eb1257fcf384beafa P b550bed5a64f466c8ba5a1761f3bef39 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 543169fd701141c181c55566d2cae00d (127.9.176.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } attrs { replace: true } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } }
I20240502 01:55:47.943459 27715 raft_consensus.cc:2932] T d404c85b11594e7eb1257fcf384beafa P d50f07af87d74f039fb2b43cf166fa43 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 543169fd701141c181c55566d2cae00d (127.9.176.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } attrs { replace: true } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } }
I20240502 01:55:47.945704 27570 catalog_manager.cc:5484] T d404c85b11594e7eb1257fcf384beafa P a64152fd9595478ca767a9f248ef3b85 reported cstate change: config changed from index 4 to 5, 543169fd701141c181c55566d2cae00d (127.9.176.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a64152fd9595478ca767a9f248ef3b85" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:55:47.946321 28778 raft_consensus.cc:2932] T d404c85b11594e7eb1257fcf384beafa P 543169fd701141c181c55566d2cae00d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 543169fd701141c181c55566d2cae00d (127.9.176.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } attrs { replace: true } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } }
I20240502 01:55:47.956116 27848 consensus_queue.cc:237] T d404c85b11594e7eb1257fcf384beafa P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } }
I20240502 01:55:47.957955 27715 raft_consensus.cc:1261] T d404c85b11594e7eb1257fcf384beafa P d50f07af87d74f039fb2b43cf166fa43 [term 1 FOLLOWER]: Refusing update from remote peer a64152fd9595478ca767a9f248ef3b85: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:47.958019 28778 raft_consensus.cc:1261] T d404c85b11594e7eb1257fcf384beafa P 543169fd701141c181c55566d2cae00d [term 1 FOLLOWER]: Refusing update from remote peer a64152fd9595478ca767a9f248ef3b85: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:47.958632 28866 consensus_queue.cc:1035] T d404c85b11594e7eb1257fcf384beafa P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:47.958899 29221 consensus_queue.cc:1035] T d404c85b11594e7eb1257fcf384beafa P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:47.961504 28956 raft_consensus.cc:2932] T d404c85b11594e7eb1257fcf384beafa P a64152fd9595478ca767a9f248ef3b85 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b550bed5a64f466c8ba5a1761f3bef39 (127.9.176.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } }
I20240502 01:55:47.962251 27715 raft_consensus.cc:2932] T d404c85b11594e7eb1257fcf384beafa P d50f07af87d74f039fb2b43cf166fa43 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b550bed5a64f466c8ba5a1761f3bef39 (127.9.176.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } }
I20240502 01:55:47.964011 28778 raft_consensus.cc:2932] T d404c85b11594e7eb1257fcf384beafa P 543169fd701141c181c55566d2cae00d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b550bed5a64f466c8ba5a1761f3bef39 (127.9.176.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } }
I20240502 01:55:47.964946 27560 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d404c85b11594e7eb1257fcf384beafa with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:55:47.967453 27572 catalog_manager.cc:5484] T d404c85b11594e7eb1257fcf384beafa P d50f07af87d74f039fb2b43cf166fa43 reported cstate change: config changed from index 5 to 6, VOTER b550bed5a64f466c8ba5a1761f3bef39 (127.9.176.195) evicted. New cstate: current_term: 1 leader_uuid: "a64152fd9595478ca767a9f248ef3b85" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } }
I20240502 01:55:47.968979 29221 raft_consensus.cc:1050] T 5e8b27dca9af473188c50269820343f1 P a64152fd9595478ca767a9f248ef3b85: attempting to promote NON_VOTER c541378ca18944c49c37f49097bc7f81 to VOTER
I20240502 01:55:47.969846 29221 consensus_queue.cc:237] T 5e8b27dca9af473188c50269820343f1 P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } attrs { replace: true } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } }
I20240502 01:55:47.972175 27962 tablet_service.cc:1514] Processing DeleteTablet for tablet d404c85b11594e7eb1257fcf384beafa with delete_type TABLET_DATA_TOMBSTONED (TS b550bed5a64f466c8ba5a1761f3bef39 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:46002
I20240502 01:55:47.973903 27983 raft_consensus.cc:1261] T 5e8b27dca9af473188c50269820343f1 P b550bed5a64f466c8ba5a1761f3bef39 [term 1 FOLLOWER]: Refusing update from remote peer a64152fd9595478ca767a9f248ef3b85: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:47.973987 28646 raft_consensus.cc:1261] T 5e8b27dca9af473188c50269820343f1 P c541378ca18944c49c37f49097bc7f81 [term 1 LEARNER]: Refusing update from remote peer a64152fd9595478ca767a9f248ef3b85: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:47.974771 28956 consensus_queue.cc:1035] T 5e8b27dca9af473188c50269820343f1 P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:47.975152 28866 consensus_queue.cc:1035] T 5e8b27dca9af473188c50269820343f1 P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:47.975203 27715 raft_consensus.cc:1261] T 5e8b27dca9af473188c50269820343f1 P d50f07af87d74f039fb2b43cf166fa43 [term 1 FOLLOWER]: Refusing update from remote peer a64152fd9595478ca767a9f248ef3b85: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:47.976290 28956 consensus_queue.cc:1035] T 5e8b27dca9af473188c50269820343f1 P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:47.982620 29231 tablet_replica.cc:330] T d404c85b11594e7eb1257fcf384beafa P b550bed5a64f466c8ba5a1761f3bef39: stopping tablet replica
I20240502 01:55:47.983646 29231 raft_consensus.cc:2229] T d404c85b11594e7eb1257fcf384beafa P b550bed5a64f466c8ba5a1761f3bef39 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:47.984583 29231 raft_consensus.cc:2258] T d404c85b11594e7eb1257fcf384beafa P b550bed5a64f466c8ba5a1761f3bef39 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:47.986600 28866 raft_consensus.cc:2932] T 5e8b27dca9af473188c50269820343f1 P a64152fd9595478ca767a9f248ef3b85 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c541378ca18944c49c37f49097bc7f81 (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } attrs { replace: true } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } }
I20240502 01:55:47.987908 29231 ts_tablet_manager.cc:1875] T d404c85b11594e7eb1257fcf384beafa P b550bed5a64f466c8ba5a1761f3bef39: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:47.991184 27983 raft_consensus.cc:2932] T 5e8b27dca9af473188c50269820343f1 P b550bed5a64f466c8ba5a1761f3bef39 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c541378ca18944c49c37f49097bc7f81 (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } attrs { replace: true } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } }
I20240502 01:55:47.992937 27566 catalog_manager.cc:5484] T 5e8b27dca9af473188c50269820343f1 P a64152fd9595478ca767a9f248ef3b85 reported cstate change: config changed from index 4 to 5, c541378ca18944c49c37f49097bc7f81 (127.9.176.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a64152fd9595478ca767a9f248ef3b85" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:55:47.993104 28646 raft_consensus.cc:2932] T 5e8b27dca9af473188c50269820343f1 P c541378ca18944c49c37f49097bc7f81 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c541378ca18944c49c37f49097bc7f81 (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } attrs { replace: true } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } }
I20240502 01:55:47.994534 29231 ts_tablet_manager.cc:1888] T d404c85b11594e7eb1257fcf384beafa P b550bed5a64f466c8ba5a1761f3bef39: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:55:47.994781 29231 log.cc:1196] T d404c85b11594e7eb1257fcf384beafa P b550bed5a64f466c8ba5a1761f3bef39: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/wal/wals/d404c85b11594e7eb1257fcf384beafa
I20240502 01:55:47.994519 27715 raft_consensus.cc:2932] T 5e8b27dca9af473188c50269820343f1 P d50f07af87d74f039fb2b43cf166fa43 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c541378ca18944c49c37f49097bc7f81 (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } attrs { replace: true } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } }
I20240502 01:55:47.996958 27560 catalog_manager.cc:4830] TS b550bed5a64f466c8ba5a1761f3bef39 (127.9.176.195:40013): tablet d404c85b11594e7eb1257fcf384beafa (table rebalance_test_table_2 [id=d490226c88e64345ad42c7ceb86bf959]) successfully deleted
I20240502 01:55:48.002310 29161 raft_consensus.cc:1050] T 9a16d465aa2b426484868a2b212c5d00 P d50f07af87d74f039fb2b43cf166fa43: attempting to promote NON_VOTER 543169fd701141c181c55566d2cae00d to VOTER
I20240502 01:55:48.003481 29161 consensus_queue.cc:237] T 9a16d465aa2b426484868a2b212c5d00 P d50f07af87d74f039fb2b43cf166fa43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } attrs { replace: true } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } }
I20240502 01:55:48.004176 27848 consensus_queue.cc:237] T 5e8b27dca9af473188c50269820343f1 P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } }
I20240502 01:55:48.007486 28646 raft_consensus.cc:1261] T 5e8b27dca9af473188c50269820343f1 P c541378ca18944c49c37f49097bc7f81 [term 1 FOLLOWER]: Refusing update from remote peer a64152fd9595478ca767a9f248ef3b85: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:48.007923 27983 raft_consensus.cc:1261] T 5e8b27dca9af473188c50269820343f1 P b550bed5a64f466c8ba5a1761f3bef39 [term 1 FOLLOWER]: Refusing update from remote peer a64152fd9595478ca767a9f248ef3b85: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:48.007993 27981 raft_consensus.cc:1261] T 9a16d465aa2b426484868a2b212c5d00 P b550bed5a64f466c8ba5a1761f3bef39 [term 1 FOLLOWER]: Refusing update from remote peer d50f07af87d74f039fb2b43cf166fa43: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:48.008577 28778 raft_consensus.cc:1261] T 9a16d465aa2b426484868a2b212c5d00 P 543169fd701141c181c55566d2cae00d [term 1 LEARNER]: Refusing update from remote peer d50f07af87d74f039fb2b43cf166fa43: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:48.009155 29160 consensus_queue.cc:1035] T 9a16d465aa2b426484868a2b212c5d00 P d50f07af87d74f039fb2b43cf166fa43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:48.009564 28977 consensus_queue.cc:1035] T 9a16d465aa2b426484868a2b212c5d00 P d50f07af87d74f039fb2b43cf166fa43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:48.009649 27848 raft_consensus.cc:1261] T 9a16d465aa2b426484868a2b212c5d00 P a64152fd9595478ca767a9f248ef3b85 [term 1 FOLLOWER]: Refusing update from remote peer d50f07af87d74f039fb2b43cf166fa43: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:48.009645 28956 consensus_queue.cc:1035] T 5e8b27dca9af473188c50269820343f1 P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:48.010300 28956 consensus_queue.cc:1035] T 5e8b27dca9af473188c50269820343f1 P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:48.010564 29160 consensus_queue.cc:1035] T 9a16d465aa2b426484868a2b212c5d00 P d50f07af87d74f039fb2b43cf166fa43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:48.012881 28956 raft_consensus.cc:2932] T 5e8b27dca9af473188c50269820343f1 P a64152fd9595478ca767a9f248ef3b85 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d50f07af87d74f039fb2b43cf166fa43 (127.9.176.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } }
I20240502 01:55:48.016110 27560 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5e8b27dca9af473188c50269820343f1 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:55:48.016819 27983 raft_consensus.cc:2932] T 5e8b27dca9af473188c50269820343f1 P b550bed5a64f466c8ba5a1761f3bef39 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d50f07af87d74f039fb2b43cf166fa43 (127.9.176.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } }
I20240502 01:55:48.018626 27566 catalog_manager.cc:5484] T 5e8b27dca9af473188c50269820343f1 P a64152fd9595478ca767a9f248ef3b85 reported cstate change: config changed from index 5 to 6, VOTER d50f07af87d74f039fb2b43cf166fa43 (127.9.176.193) evicted. New cstate: current_term: 1 leader_uuid: "a64152fd9595478ca767a9f248ef3b85" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:55:48.023453 27696 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e8b27dca9af473188c50269820343f1 with delete_type TABLET_DATA_TOMBSTONED (TS d50f07af87d74f039fb2b43cf166fa43 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35710
I20240502 01:55:48.024685 29240 tablet_replica.cc:330] T 5e8b27dca9af473188c50269820343f1 P d50f07af87d74f039fb2b43cf166fa43: stopping tablet replica
I20240502 01:55:48.025400 29240 raft_consensus.cc:2229] T 5e8b27dca9af473188c50269820343f1 P d50f07af87d74f039fb2b43cf166fa43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:48.026383 29240 raft_consensus.cc:2258] T 5e8b27dca9af473188c50269820343f1 P d50f07af87d74f039fb2b43cf166fa43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:48.025943 28646 raft_consensus.cc:2932] T 5e8b27dca9af473188c50269820343f1 P c541378ca18944c49c37f49097bc7f81 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d50f07af87d74f039fb2b43cf166fa43 (127.9.176.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } }
I20240502 01:55:48.029198 29240 ts_tablet_manager.cc:1875] T 5e8b27dca9af473188c50269820343f1 P d50f07af87d74f039fb2b43cf166fa43: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:48.034739 29240 ts_tablet_manager.cc:1888] T 5e8b27dca9af473188c50269820343f1 P d50f07af87d74f039fb2b43cf166fa43: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:55:48.034935 29240 log.cc:1196] T 5e8b27dca9af473188c50269820343f1 P d50f07af87d74f039fb2b43cf166fa43: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/wal/wals/5e8b27dca9af473188c50269820343f1
I20240502 01:55:48.035940 27559 catalog_manager.cc:4830] TS d50f07af87d74f039fb2b43cf166fa43 (127.9.176.193:34203): tablet 5e8b27dca9af473188c50269820343f1 (table rebalance_test_table_1 [id=fdfd057dc2884c01bb7ee4631f5e3566]) successfully deleted
I20240502 01:55:48.037698 29160 raft_consensus.cc:2932] T 9a16d465aa2b426484868a2b212c5d00 P d50f07af87d74f039fb2b43cf166fa43 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 543169fd701141c181c55566d2cae00d (127.9.176.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } attrs { replace: true } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } }
I20240502 01:55:48.039180 28778 raft_consensus.cc:2932] T 9a16d465aa2b426484868a2b212c5d00 P 543169fd701141c181c55566d2cae00d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 543169fd701141c181c55566d2cae00d (127.9.176.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } attrs { replace: true } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } }
I20240502 01:55:48.041975 27849 raft_consensus.cc:2932] T 9a16d465aa2b426484868a2b212c5d00 P a64152fd9595478ca767a9f248ef3b85 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 543169fd701141c181c55566d2cae00d (127.9.176.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } attrs { replace: true } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } }
I20240502 01:55:48.042539 27566 catalog_manager.cc:5484] T 9a16d465aa2b426484868a2b212c5d00 P d50f07af87d74f039fb2b43cf166fa43 reported cstate change: config changed from index 4 to 5, 543169fd701141c181c55566d2cae00d (127.9.176.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d50f07af87d74f039fb2b43cf166fa43" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:55:48.046567 27981 raft_consensus.cc:2932] T 9a16d465aa2b426484868a2b212c5d00 P b550bed5a64f466c8ba5a1761f3bef39 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 543169fd701141c181c55566d2cae00d (127.9.176.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } attrs { replace: true } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } }
I20240502 01:55:48.051033 27715 consensus_queue.cc:237] T 9a16d465aa2b426484868a2b212c5d00 P d50f07af87d74f039fb2b43cf166fa43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } }
I20240502 01:55:48.053180 28778 raft_consensus.cc:1261] T 9a16d465aa2b426484868a2b212c5d00 P 543169fd701141c181c55566d2cae00d [term 1 FOLLOWER]: Refusing update from remote peer d50f07af87d74f039fb2b43cf166fa43: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:48.053365 27849 raft_consensus.cc:1261] T 9a16d465aa2b426484868a2b212c5d00 P a64152fd9595478ca767a9f248ef3b85 [term 1 FOLLOWER]: Refusing update from remote peer d50f07af87d74f039fb2b43cf166fa43: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:48.053908 28977 consensus_queue.cc:1035] T 9a16d465aa2b426484868a2b212c5d00 P d50f07af87d74f039fb2b43cf166fa43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:48.054219 29160 consensus_queue.cc:1035] T 9a16d465aa2b426484868a2b212c5d00 P d50f07af87d74f039fb2b43cf166fa43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:48.056950 28977 raft_consensus.cc:2932] T 9a16d465aa2b426484868a2b212c5d00 P d50f07af87d74f039fb2b43cf166fa43 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b550bed5a64f466c8ba5a1761f3bef39 (127.9.176.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } }
I20240502 01:55:48.060406 27849 raft_consensus.cc:2932] T 9a16d465aa2b426484868a2b212c5d00 P a64152fd9595478ca767a9f248ef3b85 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b550bed5a64f466c8ba5a1761f3bef39 (127.9.176.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } }
I20240502 01:55:48.061523 27559 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9a16d465aa2b426484868a2b212c5d00 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:55:48.065960 27570 catalog_manager.cc:5484] T 9a16d465aa2b426484868a2b212c5d00 P d50f07af87d74f039fb2b43cf166fa43 reported cstate change: config changed from index 5 to 6, VOTER b550bed5a64f466c8ba5a1761f3bef39 (127.9.176.195) evicted. New cstate: current_term: 1 leader_uuid: "d50f07af87d74f039fb2b43cf166fa43" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:55:48.066170 28778 raft_consensus.cc:2932] T 9a16d465aa2b426484868a2b212c5d00 P 543169fd701141c181c55566d2cae00d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b550bed5a64f466c8ba5a1761f3bef39 (127.9.176.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } }
I20240502 01:55:48.070259 27962 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a16d465aa2b426484868a2b212c5d00 with delete_type TABLET_DATA_TOMBSTONED (TS b550bed5a64f466c8ba5a1761f3bef39 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:46002
I20240502 01:55:48.070712 29231 tablet_replica.cc:330] T 9a16d465aa2b426484868a2b212c5d00 P b550bed5a64f466c8ba5a1761f3bef39: stopping tablet replica
I20240502 01:55:48.070951 29231 raft_consensus.cc:2229] T 9a16d465aa2b426484868a2b212c5d00 P b550bed5a64f466c8ba5a1761f3bef39 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:48.071333 29231 raft_consensus.cc:2258] T 9a16d465aa2b426484868a2b212c5d00 P b550bed5a64f466c8ba5a1761f3bef39 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:48.072489 29231 ts_tablet_manager.cc:1875] T 9a16d465aa2b426484868a2b212c5d00 P b550bed5a64f466c8ba5a1761f3bef39: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:48.078589 29231 ts_tablet_manager.cc:1888] T 9a16d465aa2b426484868a2b212c5d00 P b550bed5a64f466c8ba5a1761f3bef39: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:55:48.078897 29231 log.cc:1196] T 9a16d465aa2b426484868a2b212c5d00 P b550bed5a64f466c8ba5a1761f3bef39: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/wal/wals/9a16d465aa2b426484868a2b212c5d00
I20240502 01:55:48.079885 27560 catalog_manager.cc:4830] TS b550bed5a64f466c8ba5a1761f3bef39 (127.9.176.195:40013): tablet 9a16d465aa2b426484868a2b212c5d00 (table rebalance_test_table_0 [id=8ef52b859d42498b92b6b2ec0805663b]) successfully deleted
I20240502 01:55:48.100790 29221 raft_consensus.cc:1050] T 31de6c641c094cce894d8513b878350c P a64152fd9595478ca767a9f248ef3b85: attempting to promote NON_VOTER c541378ca18944c49c37f49097bc7f81 to VOTER
I20240502 01:55:48.101553 29221 consensus_queue.cc:237] T 31de6c641c094cce894d8513b878350c P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } }
I20240502 01:55:48.104455 27981 raft_consensus.cc:1261] T 31de6c641c094cce894d8513b878350c P b550bed5a64f466c8ba5a1761f3bef39 [term 1 FOLLOWER]: Refusing update from remote peer a64152fd9595478ca767a9f248ef3b85: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:48.104574 28646 raft_consensus.cc:1261] T 31de6c641c094cce894d8513b878350c P c541378ca18944c49c37f49097bc7f81 [term 1 LEARNER]: Refusing update from remote peer a64152fd9595478ca767a9f248ef3b85: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:48.104835 27715 raft_consensus.cc:1261] T 31de6c641c094cce894d8513b878350c P d50f07af87d74f039fb2b43cf166fa43 [term 1 FOLLOWER]: Refusing update from remote peer a64152fd9595478ca767a9f248ef3b85: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:48.105355 29221 consensus_queue.cc:1035] T 31de6c641c094cce894d8513b878350c P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:48.105717 28866 consensus_queue.cc:1035] T 31de6c641c094cce894d8513b878350c P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:48.106025 28956 consensus_queue.cc:1035] T 31de6c641c094cce894d8513b878350c P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:48.109766 28866 raft_consensus.cc:2932] T 31de6c641c094cce894d8513b878350c P a64152fd9595478ca767a9f248ef3b85 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c541378ca18944c49c37f49097bc7f81 (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } }
I20240502 01:55:48.110859 27715 raft_consensus.cc:2932] T 31de6c641c094cce894d8513b878350c P d50f07af87d74f039fb2b43cf166fa43 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c541378ca18944c49c37f49097bc7f81 (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } }
I20240502 01:55:48.115123 28646 raft_consensus.cc:2932] T 31de6c641c094cce894d8513b878350c P c541378ca18944c49c37f49097bc7f81 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c541378ca18944c49c37f49097bc7f81 (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } }
I20240502 01:55:48.115705 27566 catalog_manager.cc:5484] T 31de6c641c094cce894d8513b878350c P d50f07af87d74f039fb2b43cf166fa43 reported cstate change: config changed from index 4 to 5, c541378ca18944c49c37f49097bc7f81 (127.9.176.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a64152fd9595478ca767a9f248ef3b85" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } }
I20240502 01:55:48.118607 27981 raft_consensus.cc:2932] T 31de6c641c094cce894d8513b878350c P b550bed5a64f466c8ba5a1761f3bef39 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c541378ca18944c49c37f49097bc7f81 (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } }
I20240502 01:55:48.129065 27849 consensus_queue.cc:237] T 31de6c641c094cce894d8513b878350c P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } }
I20240502 01:55:48.131484 28646 raft_consensus.cc:1261] T 31de6c641c094cce894d8513b878350c P c541378ca18944c49c37f49097bc7f81 [term 1 FOLLOWER]: Refusing update from remote peer a64152fd9595478ca767a9f248ef3b85: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:48.131652 27981 raft_consensus.cc:1261] T 31de6c641c094cce894d8513b878350c P b550bed5a64f466c8ba5a1761f3bef39 [term 1 FOLLOWER]: Refusing update from remote peer a64152fd9595478ca767a9f248ef3b85: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:48.132649 29161 raft_consensus.cc:1050] T f22c0dfa05664a4a9f10aa81424a396e P d50f07af87d74f039fb2b43cf166fa43: attempting to promote NON_VOTER 543169fd701141c181c55566d2cae00d to VOTER
I20240502 01:55:48.132624 28866 consensus_queue.cc:1035] T 31de6c641c094cce894d8513b878350c P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:48.133283 28866 consensus_queue.cc:1035] T 31de6c641c094cce894d8513b878350c P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:48.133468 29161 consensus_queue.cc:237] T f22c0dfa05664a4a9f10aa81424a396e P d50f07af87d74f039fb2b43cf166fa43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } attrs { replace: true } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } }
I20240502 01:55:48.135334 28866 raft_consensus.cc:2932] T 31de6c641c094cce894d8513b878350c P a64152fd9595478ca767a9f248ef3b85 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d50f07af87d74f039fb2b43cf166fa43 (127.9.176.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } }
I20240502 01:55:48.136718 27983 raft_consensus.cc:1261] T f22c0dfa05664a4a9f10aa81424a396e P b550bed5a64f466c8ba5a1761f3bef39 [term 1 FOLLOWER]: Refusing update from remote peer d50f07af87d74f039fb2b43cf166fa43: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:48.136768 27981 raft_consensus.cc:2932] T 31de6c641c094cce894d8513b878350c P b550bed5a64f466c8ba5a1761f3bef39 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d50f07af87d74f039fb2b43cf166fa43 (127.9.176.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } }
I20240502 01:55:48.137487 27849 raft_consensus.cc:1261] T f22c0dfa05664a4a9f10aa81424a396e P a64152fd9595478ca767a9f248ef3b85 [term 1 FOLLOWER]: Refusing update from remote peer d50f07af87d74f039fb2b43cf166fa43: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:48.137518 29161 consensus_queue.cc:1035] T f22c0dfa05664a4a9f10aa81424a396e P d50f07af87d74f039fb2b43cf166fa43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:48.138748 29161 consensus_queue.cc:1035] T f22c0dfa05664a4a9f10aa81424a396e P d50f07af87d74f039fb2b43cf166fa43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:48.139539 28778 raft_consensus.cc:1261] T f22c0dfa05664a4a9f10aa81424a396e P 543169fd701141c181c55566d2cae00d [term 1 LEARNER]: Refusing update from remote peer d50f07af87d74f039fb2b43cf166fa43: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:48.140151 29161 consensus_queue.cc:1035] T f22c0dfa05664a4a9f10aa81424a396e P d50f07af87d74f039fb2b43cf166fa43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:48.140316 27560 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 31de6c641c094cce894d8513b878350c with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:55:48.142378 27570 catalog_manager.cc:5484] T 31de6c641c094cce894d8513b878350c P a64152fd9595478ca767a9f248ef3b85 reported cstate change: config changed from index 5 to 6, VOTER d50f07af87d74f039fb2b43cf166fa43 (127.9.176.193) evicted. New cstate: current_term: 1 leader_uuid: "a64152fd9595478ca767a9f248ef3b85" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:55:48.142287 28646 raft_consensus.cc:2932] T 31de6c641c094cce894d8513b878350c P c541378ca18944c49c37f49097bc7f81 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d50f07af87d74f039fb2b43cf166fa43 (127.9.176.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } }
I20240502 01:55:48.146845 27696 tablet_service.cc:1514] Processing DeleteTablet for tablet 31de6c641c094cce894d8513b878350c with delete_type TABLET_DATA_TOMBSTONED (TS d50f07af87d74f039fb2b43cf166fa43 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35710
I20240502 01:55:48.147264 29240 tablet_replica.cc:330] T 31de6c641c094cce894d8513b878350c P d50f07af87d74f039fb2b43cf166fa43: stopping tablet replica
I20240502 01:55:48.147536 29240 raft_consensus.cc:2229] T 31de6c641c094cce894d8513b878350c P d50f07af87d74f039fb2b43cf166fa43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:48.147967 29240 raft_consensus.cc:2258] T 31de6c641c094cce894d8513b878350c P d50f07af87d74f039fb2b43cf166fa43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:48.149060 29240 ts_tablet_manager.cc:1875] T 31de6c641c094cce894d8513b878350c P d50f07af87d74f039fb2b43cf166fa43: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:48.153028 29237 raft_consensus.cc:2932] T f22c0dfa05664a4a9f10aa81424a396e P d50f07af87d74f039fb2b43cf166fa43 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 543169fd701141c181c55566d2cae00d (127.9.176.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } attrs { replace: true } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } }
I20240502 01:55:48.154047 27983 raft_consensus.cc:2932] T f22c0dfa05664a4a9f10aa81424a396e P b550bed5a64f466c8ba5a1761f3bef39 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 543169fd701141c181c55566d2cae00d (127.9.176.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } attrs { replace: true } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } }
I20240502 01:55:48.156358 28866 raft_consensus.cc:1050] T f1597c56a1a74b37bb5e18e8c4662f46 P a64152fd9595478ca767a9f248ef3b85: attempting to promote NON_VOTER c541378ca18944c49c37f49097bc7f81 to VOTER
I20240502 01:55:48.157282 28866 consensus_queue.cc:237] T f1597c56a1a74b37bb5e18e8c4662f46 P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } }
I20240502 01:55:48.160732 27566 catalog_manager.cc:5484] T f22c0dfa05664a4a9f10aa81424a396e P b550bed5a64f466c8ba5a1761f3bef39 reported cstate change: config changed from index 4 to 5, 543169fd701141c181c55566d2cae00d (127.9.176.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d50f07af87d74f039fb2b43cf166fa43" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } attrs { replace: true } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } }
I20240502 01:55:48.164916 28778 raft_consensus.cc:2932] T f22c0dfa05664a4a9f10aa81424a396e P 543169fd701141c181c55566d2cae00d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 543169fd701141c181c55566d2cae00d (127.9.176.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } attrs { replace: true } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } }
I20240502 01:55:48.180359 29240 ts_tablet_manager.cc:1888] T 31de6c641c094cce894d8513b878350c P d50f07af87d74f039fb2b43cf166fa43: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:55:48.182298 28645 raft_consensus.cc:1261] T f1597c56a1a74b37bb5e18e8c4662f46 P c541378ca18944c49c37f49097bc7f81 [term 1 LEARNER]: Refusing update from remote peer a64152fd9595478ca767a9f248ef3b85: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:48.182390 27983 raft_consensus.cc:1261] T f1597c56a1a74b37bb5e18e8c4662f46 P b550bed5a64f466c8ba5a1761f3bef39 [term 1 FOLLOWER]: Refusing update from remote peer a64152fd9595478ca767a9f248ef3b85: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:48.183331 28866 consensus_queue.cc:1035] T f1597c56a1a74b37bb5e18e8c4662f46 P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:48.183858 28956 consensus_queue.cc:1035] T f1597c56a1a74b37bb5e18e8c4662f46 P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:48.182390 27715 raft_consensus.cc:1261] T f1597c56a1a74b37bb5e18e8c4662f46 P d50f07af87d74f039fb2b43cf166fa43 [term 1 FOLLOWER]: Refusing update from remote peer a64152fd9595478ca767a9f248ef3b85: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:48.185274 29240 log.cc:1196] T 31de6c641c094cce894d8513b878350c P d50f07af87d74f039fb2b43cf166fa43: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/wal/wals/31de6c641c094cce894d8513b878350c
I20240502 01:55:48.186362 28866 consensus_queue.cc:1035] T f1597c56a1a74b37bb5e18e8c4662f46 P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:48.187839 27559 catalog_manager.cc:4830] TS d50f07af87d74f039fb2b43cf166fa43 (127.9.176.193:34203): tablet 31de6c641c094cce894d8513b878350c (table rebalance_test_table_2 [id=d490226c88e64345ad42c7ceb86bf959]) successfully deleted
I20240502 01:55:48.189194 27849 raft_consensus.cc:2932] T f22c0dfa05664a4a9f10aa81424a396e P a64152fd9595478ca767a9f248ef3b85 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 543169fd701141c181c55566d2cae00d (127.9.176.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } attrs { replace: true } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } }
I20240502 01:55:48.192785 27716 consensus_queue.cc:237] T f22c0dfa05664a4a9f10aa81424a396e P d50f07af87d74f039fb2b43cf166fa43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } }
I20240502 01:55:48.196496 28778 raft_consensus.cc:1261] T f22c0dfa05664a4a9f10aa81424a396e P 543169fd701141c181c55566d2cae00d [term 1 FOLLOWER]: Refusing update from remote peer d50f07af87d74f039fb2b43cf166fa43: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:48.197563 29237 consensus_queue.cc:1035] T f22c0dfa05664a4a9f10aa81424a396e P d50f07af87d74f039fb2b43cf166fa43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:48.197839 27848 raft_consensus.cc:1261] T f22c0dfa05664a4a9f10aa81424a396e P a64152fd9595478ca767a9f248ef3b85 [term 1 FOLLOWER]: Refusing update from remote peer d50f07af87d74f039fb2b43cf166fa43: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:48.201223 29237 consensus_queue.cc:1035] T f22c0dfa05664a4a9f10aa81424a396e P d50f07af87d74f039fb2b43cf166fa43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:48.203009 28956 raft_consensus.cc:2932] T f1597c56a1a74b37bb5e18e8c4662f46 P a64152fd9595478ca767a9f248ef3b85 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c541378ca18944c49c37f49097bc7f81 (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } }
I20240502 01:55:48.205583 29237 raft_consensus.cc:2932] T f22c0dfa05664a4a9f10aa81424a396e P d50f07af87d74f039fb2b43cf166fa43 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b550bed5a64f466c8ba5a1761f3bef39 (127.9.176.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } }
I20240502 01:55:48.206490 27849 raft_consensus.cc:2932] T f22c0dfa05664a4a9f10aa81424a396e P a64152fd9595478ca767a9f248ef3b85 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b550bed5a64f466c8ba5a1761f3bef39 (127.9.176.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } }
I20240502 01:55:48.208552 27981 raft_consensus.cc:2932] T f1597c56a1a74b37bb5e18e8c4662f46 P b550bed5a64f466c8ba5a1761f3bef39 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c541378ca18944c49c37f49097bc7f81 (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } }
I20240502 01:55:48.208894 27715 raft_consensus.cc:2932] T f1597c56a1a74b37bb5e18e8c4662f46 P d50f07af87d74f039fb2b43cf166fa43 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c541378ca18944c49c37f49097bc7f81 (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } }
I20240502 01:55:48.211637 27566 catalog_manager.cc:5484] T f1597c56a1a74b37bb5e18e8c4662f46 P a64152fd9595478ca767a9f248ef3b85 reported cstate change: config changed from index 4 to 5, c541378ca18944c49c37f49097bc7f81 (127.9.176.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a64152fd9595478ca767a9f248ef3b85" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:55:48.216336 28778 raft_consensus.cc:2932] T f22c0dfa05664a4a9f10aa81424a396e P 543169fd701141c181c55566d2cae00d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b550bed5a64f466c8ba5a1761f3bef39 (127.9.176.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } }
I20240502 01:55:48.218829 27559 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f22c0dfa05664a4a9f10aa81424a396e with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:55:48.221881 27572 catalog_manager.cc:5484] T f22c0dfa05664a4a9f10aa81424a396e P 543169fd701141c181c55566d2cae00d reported cstate change: config changed from index 5 to 6, VOTER b550bed5a64f466c8ba5a1761f3bef39 (127.9.176.195) evicted. New cstate: current_term: 1 leader_uuid: "d50f07af87d74f039fb2b43cf166fa43" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } }
I20240502 01:55:48.226445 27962 tablet_service.cc:1514] Processing DeleteTablet for tablet f22c0dfa05664a4a9f10aa81424a396e with delete_type TABLET_DATA_TOMBSTONED (TS b550bed5a64f466c8ba5a1761f3bef39 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:46002
I20240502 01:55:48.226903 29231 tablet_replica.cc:330] T f22c0dfa05664a4a9f10aa81424a396e P b550bed5a64f466c8ba5a1761f3bef39: stopping tablet replica
I20240502 01:55:48.227145 29231 raft_consensus.cc:2229] T f22c0dfa05664a4a9f10aa81424a396e P b550bed5a64f466c8ba5a1761f3bef39 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:48.227607 29231 raft_consensus.cc:2258] T f22c0dfa05664a4a9f10aa81424a396e P b550bed5a64f466c8ba5a1761f3bef39 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:48.228775 29231 ts_tablet_manager.cc:1875] T f22c0dfa05664a4a9f10aa81424a396e P b550bed5a64f466c8ba5a1761f3bef39: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:48.229359 28646 raft_consensus.cc:2932] T f1597c56a1a74b37bb5e18e8c4662f46 P c541378ca18944c49c37f49097bc7f81 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c541378ca18944c49c37f49097bc7f81 (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } }
I20240502 01:55:48.233206 27849 consensus_queue.cc:237] T f1597c56a1a74b37bb5e18e8c4662f46 P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } }
I20240502 01:55:48.235787 27981 raft_consensus.cc:1261] T f1597c56a1a74b37bb5e18e8c4662f46 P b550bed5a64f466c8ba5a1761f3bef39 [term 1 FOLLOWER]: Refusing update from remote peer a64152fd9595478ca767a9f248ef3b85: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:48.236364 29221 consensus_queue.cc:1035] T f1597c56a1a74b37bb5e18e8c4662f46 P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:48.237165 28645 raft_consensus.cc:1261] T f1597c56a1a74b37bb5e18e8c4662f46 P c541378ca18944c49c37f49097bc7f81 [term 1 FOLLOWER]: Refusing update from remote peer a64152fd9595478ca767a9f248ef3b85: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:48.238983 29221 consensus_queue.cc:1035] T f1597c56a1a74b37bb5e18e8c4662f46 P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:48.239244 28956 raft_consensus.cc:2932] T f1597c56a1a74b37bb5e18e8c4662f46 P a64152fd9595478ca767a9f248ef3b85 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d50f07af87d74f039fb2b43cf166fa43 (127.9.176.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } }
I20240502 01:55:48.240172 27981 raft_consensus.cc:2932] T f1597c56a1a74b37bb5e18e8c4662f46 P b550bed5a64f466c8ba5a1761f3bef39 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d50f07af87d74f039fb2b43cf166fa43 (127.9.176.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } }
I20240502 01:55:48.243882 27560 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f1597c56a1a74b37bb5e18e8c4662f46 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:55:48.244392 28646 raft_consensus.cc:2932] T f1597c56a1a74b37bb5e18e8c4662f46 P c541378ca18944c49c37f49097bc7f81 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d50f07af87d74f039fb2b43cf166fa43 (127.9.176.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } }
I20240502 01:55:48.246296 27566 catalog_manager.cc:5484] T f1597c56a1a74b37bb5e18e8c4662f46 P b550bed5a64f466c8ba5a1761f3bef39 reported cstate change: config changed from index 5 to 6, VOTER d50f07af87d74f039fb2b43cf166fa43 (127.9.176.193) evicted. New cstate: current_term: 1 leader_uuid: "a64152fd9595478ca767a9f248ef3b85" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } }
I20240502 01:55:48.247797 29231 ts_tablet_manager.cc:1888] T f22c0dfa05664a4a9f10aa81424a396e P b550bed5a64f466c8ba5a1761f3bef39: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:55:48.247946 29231 log.cc:1196] T f22c0dfa05664a4a9f10aa81424a396e P b550bed5a64f466c8ba5a1761f3bef39: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-2/wal/wals/f22c0dfa05664a4a9f10aa81424a396e
I20240502 01:55:48.248850 27560 catalog_manager.cc:4830] TS b550bed5a64f466c8ba5a1761f3bef39 (127.9.176.195:40013): tablet f22c0dfa05664a4a9f10aa81424a396e (table rebalance_test_table_1 [id=fdfd057dc2884c01bb7ee4631f5e3566]) successfully deleted
I20240502 01:55:48.253178 27696 tablet_service.cc:1514] Processing DeleteTablet for tablet f1597c56a1a74b37bb5e18e8c4662f46 with delete_type TABLET_DATA_TOMBSTONED (TS d50f07af87d74f039fb2b43cf166fa43 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35710
I20240502 01:55:48.253661 29240 tablet_replica.cc:330] T f1597c56a1a74b37bb5e18e8c4662f46 P d50f07af87d74f039fb2b43cf166fa43: stopping tablet replica
I20240502 01:55:48.253908 29240 raft_consensus.cc:2229] T f1597c56a1a74b37bb5e18e8c4662f46 P d50f07af87d74f039fb2b43cf166fa43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:48.254236 29240 raft_consensus.cc:2258] T f1597c56a1a74b37bb5e18e8c4662f46 P d50f07af87d74f039fb2b43cf166fa43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:48.255538 29240 ts_tablet_manager.cc:1875] T f1597c56a1a74b37bb5e18e8c4662f46 P d50f07af87d74f039fb2b43cf166fa43: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:48.262343 29240 ts_tablet_manager.cc:1888] T f1597c56a1a74b37bb5e18e8c4662f46 P d50f07af87d74f039fb2b43cf166fa43: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:55:48.262544 29240 log.cc:1196] T f1597c56a1a74b37bb5e18e8c4662f46 P d50f07af87d74f039fb2b43cf166fa43: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-0/wal/wals/f1597c56a1a74b37bb5e18e8c4662f46
I20240502 01:55:48.263424 27559 catalog_manager.cc:4830] TS d50f07af87d74f039fb2b43cf166fa43 (127.9.176.193:34203): tablet f1597c56a1a74b37bb5e18e8c4662f46 (table rebalance_test_table_0 [id=8ef52b859d42498b92b6b2ec0805663b]) successfully deleted
I20240502 01:55:48.394358 28758 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:48.397832 27827 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:55:48.398427 27696 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:55:48.406172 27962 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:48.407853 28626 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:48.583165 28758 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:48.608237 27696 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:55:48.614658 27827 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:55:48.615167 28626 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:48.615602 27962 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:48.804956 28758 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:48.809994 27827 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:55:48.812381 28626 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:48.820717 27696 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:55:48.822420 27962 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:49.467103 27566 ts_manager.cc:292] Set tserver state for a64152fd9595478ca767a9f248ef3b85 to MAINTENANCE_MODE
I20240502 01:55:51.043004 28758 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:51.062323 27962 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:51.072798 27827 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:55:51.083297 28626 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:51.094373 27696 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:55:51.268172 28758 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:51.268810 27827 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:55:51.278090 28626 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:51.289212 27962 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:51.307796 27696 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:55:51.503741 27827 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:55:51.509223 28758 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:51.526773 27696 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:55:51.527822 27962 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:51.529371 28626 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:51.614104 27849 consensus_queue.cc:237] T f1597c56a1a74b37bb5e18e8c4662f46 P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } }
I20240502 01:55:51.616351 27983 raft_consensus.cc:1261] T f1597c56a1a74b37bb5e18e8c4662f46 P b550bed5a64f466c8ba5a1761f3bef39 [term 1 FOLLOWER]: Refusing update from remote peer a64152fd9595478ca767a9f248ef3b85: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:51.616426 28646 raft_consensus.cc:1261] T f1597c56a1a74b37bb5e18e8c4662f46 P c541378ca18944c49c37f49097bc7f81 [term 1 FOLLOWER]: Refusing update from remote peer a64152fd9595478ca767a9f248ef3b85: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:51.617039 29391 consensus_queue.cc:1035] T f1597c56a1a74b37bb5e18e8c4662f46 P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:51.617394 29516 consensus_queue.cc:1035] T f1597c56a1a74b37bb5e18e8c4662f46 P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:51.620591 29516 raft_consensus.cc:2932] T f1597c56a1a74b37bb5e18e8c4662f46 P a64152fd9595478ca767a9f248ef3b85 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } }
I20240502 01:55:51.621074 28646 raft_consensus.cc:2932] T f1597c56a1a74b37bb5e18e8c4662f46 P c541378ca18944c49c37f49097bc7f81 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } }
I20240502 01:55:51.621301 27983 raft_consensus.cc:2932] T f1597c56a1a74b37bb5e18e8c4662f46 P b550bed5a64f466c8ba5a1761f3bef39 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } }
I20240502 01:55:51.627179 27566 catalog_manager.cc:5484] T f1597c56a1a74b37bb5e18e8c4662f46 P b550bed5a64f466c8ba5a1761f3bef39 reported cstate change: config changed from index 6 to 7. New cstate: current_term: 1 leader_uuid: "a64152fd9595478ca767a9f248ef3b85" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } }
I20240502 01:55:51.636202 27849 consensus_queue.cc:237] T f1597c56a1a74b37bb5e18e8c4662f46 P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: true } }
W20240502 01:55:51.639256 27782 consensus_peers.cc:485] T f1597c56a1a74b37bb5e18e8c4662f46 P a64152fd9595478ca767a9f248ef3b85 -> Peer 543169fd701141c181c55566d2cae00d (127.9.176.196:38773): Couldn't send request to peer 543169fd701141c181c55566d2cae00d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f1597c56a1a74b37bb5e18e8c4662f46. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:51.640117 27981 raft_consensus.cc:1261] T f1597c56a1a74b37bb5e18e8c4662f46 P b550bed5a64f466c8ba5a1761f3bef39 [term 1 FOLLOWER]: Refusing update from remote peer a64152fd9595478ca767a9f248ef3b85: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:51.640188 28646 raft_consensus.cc:1261] T f1597c56a1a74b37bb5e18e8c4662f46 P c541378ca18944c49c37f49097bc7f81 [term 1 FOLLOWER]: Refusing update from remote peer a64152fd9595478ca767a9f248ef3b85: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:51.640412 27848 consensus_queue.cc:237] T 31de6c641c094cce894d8513b878350c P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } }
I20240502 01:55:51.640838 29537 consensus_queue.cc:1035] T f1597c56a1a74b37bb5e18e8c4662f46 P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:51.641157 29516 consensus_queue.cc:1035] T f1597c56a1a74b37bb5e18e8c4662f46 P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:51.643528 29525 raft_consensus.cc:2932] T f1597c56a1a74b37bb5e18e8c4662f46 P a64152fd9595478ca767a9f248ef3b85 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 543169fd701141c181c55566d2cae00d (127.9.176.196) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: true } } }
I20240502 01:55:51.644147 28646 raft_consensus.cc:2932] T f1597c56a1a74b37bb5e18e8c4662f46 P c541378ca18944c49c37f49097bc7f81 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 543169fd701141c181c55566d2cae00d (127.9.176.196) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: true } } }
I20240502 01:55:51.649837 27981 raft_consensus.cc:2932] T f1597c56a1a74b37bb5e18e8c4662f46 P b550bed5a64f466c8ba5a1761f3bef39 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 543169fd701141c181c55566d2cae00d (127.9.176.196) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: true } } }
I20240502 01:55:51.650146 27572 catalog_manager.cc:5484] T f1597c56a1a74b37bb5e18e8c4662f46 P c541378ca18944c49c37f49097bc7f81 reported cstate change: config changed from index 7 to 8, NON_VOTER 543169fd701141c181c55566d2cae00d (127.9.176.196) added. New cstate: current_term: 1 leader_uuid: "a64152fd9595478ca767a9f248ef3b85" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: true } } }
I20240502 01:55:51.652537 28646 raft_consensus.cc:1261] T 31de6c641c094cce894d8513b878350c P c541378ca18944c49c37f49097bc7f81 [term 1 FOLLOWER]: Refusing update from remote peer a64152fd9595478ca767a9f248ef3b85: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:51.652786 27983 raft_consensus.cc:1261] T 31de6c641c094cce894d8513b878350c P b550bed5a64f466c8ba5a1761f3bef39 [term 1 FOLLOWER]: Refusing update from remote peer a64152fd9595478ca767a9f248ef3b85: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:51.652815 27560 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet f1597c56a1a74b37bb5e18e8c4662f46 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 01:55:51.653450 29391 consensus_queue.cc:1035] T 31de6c641c094cce894d8513b878350c P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:51.653844 29525 consensus_queue.cc:1035] T 31de6c641c094cce894d8513b878350c P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:51.657745 29391 raft_consensus.cc:2932] T 31de6c641c094cce894d8513b878350c P a64152fd9595478ca767a9f248ef3b85 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } }
I20240502 01:55:51.660573 28645 raft_consensus.cc:2932] T 31de6c641c094cce894d8513b878350c P c541378ca18944c49c37f49097bc7f81 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } }
I20240502 01:55:51.671828 27566 catalog_manager.cc:5484] T 31de6c641c094cce894d8513b878350c P c541378ca18944c49c37f49097bc7f81 reported cstate change: config changed from index 6 to 7. New cstate: current_term: 1 leader_uuid: "a64152fd9595478ca767a9f248ef3b85" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } }
I20240502 01:55:51.681994 27983 raft_consensus.cc:2932] T 31de6c641c094cce894d8513b878350c P b550bed5a64f466c8ba5a1761f3bef39 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } }
I20240502 01:55:51.684823 27848 consensus_queue.cc:237] T 31de6c641c094cce894d8513b878350c P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: true } }
W20240502 01:55:51.688283 27782 consensus_peers.cc:485] T 31de6c641c094cce894d8513b878350c P a64152fd9595478ca767a9f248ef3b85 -> Peer 543169fd701141c181c55566d2cae00d (127.9.176.196:38773): Couldn't send request to peer 543169fd701141c181c55566d2cae00d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 31de6c641c094cce894d8513b878350c. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:51.689536 28645 raft_consensus.cc:1261] T 31de6c641c094cce894d8513b878350c P c541378ca18944c49c37f49097bc7f81 [term 1 FOLLOWER]: Refusing update from remote peer a64152fd9595478ca767a9f248ef3b85: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:51.689877 27981 raft_consensus.cc:1261] T 31de6c641c094cce894d8513b878350c P b550bed5a64f466c8ba5a1761f3bef39 [term 1 FOLLOWER]: Refusing update from remote peer a64152fd9595478ca767a9f248ef3b85: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:51.690829 29539 consensus_queue.cc:1035] T 31de6c641c094cce894d8513b878350c P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:51.691192 29537 consensus_queue.cc:1035] T 31de6c641c094cce894d8513b878350c P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:51.694434 29391 raft_consensus.cc:2932] T 31de6c641c094cce894d8513b878350c P a64152fd9595478ca767a9f248ef3b85 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 543169fd701141c181c55566d2cae00d (127.9.176.196) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: true } } }
I20240502 01:55:51.696444 27981 raft_consensus.cc:2932] T 31de6c641c094cce894d8513b878350c P b550bed5a64f466c8ba5a1761f3bef39 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 543169fd701141c181c55566d2cae00d (127.9.176.196) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: true } } }
I20240502 01:55:51.697512 28645 raft_consensus.cc:2932] T 31de6c641c094cce894d8513b878350c P c541378ca18944c49c37f49097bc7f81 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 543169fd701141c181c55566d2cae00d (127.9.176.196) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: true } } }
I20240502 01:55:51.699297 27560 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 31de6c641c094cce894d8513b878350c with cas_config_opid_index 7: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 01:55:51.701139 27566 catalog_manager.cc:5484] T 31de6c641c094cce894d8513b878350c P a64152fd9595478ca767a9f248ef3b85 reported cstate change: config changed from index 7 to 8, NON_VOTER 543169fd701141c181c55566d2cae00d (127.9.176.196) added. New cstate: current_term: 1 leader_uuid: "a64152fd9595478ca767a9f248ef3b85" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:55:51.714336 27848 consensus_queue.cc:237] T d404c85b11594e7eb1257fcf384beafa P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } }
I20240502 01:55:51.716953 27715 raft_consensus.cc:1261] T d404c85b11594e7eb1257fcf384beafa P d50f07af87d74f039fb2b43cf166fa43 [term 1 FOLLOWER]: Refusing update from remote peer a64152fd9595478ca767a9f248ef3b85: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:51.717080 28778 raft_consensus.cc:1261] T d404c85b11594e7eb1257fcf384beafa P 543169fd701141c181c55566d2cae00d [term 1 FOLLOWER]: Refusing update from remote peer a64152fd9595478ca767a9f248ef3b85: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:51.717694 29539 consensus_queue.cc:1035] T d404c85b11594e7eb1257fcf384beafa P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:51.718071 29516 consensus_queue.cc:1035] T d404c85b11594e7eb1257fcf384beafa P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:51.721365 29539 raft_consensus.cc:2932] T d404c85b11594e7eb1257fcf384beafa P a64152fd9595478ca767a9f248ef3b85 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } }
I20240502 01:55:51.724251 27715 raft_consensus.cc:2932] T d404c85b11594e7eb1257fcf384beafa P d50f07af87d74f039fb2b43cf166fa43 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } }
I20240502 01:55:51.726770 27570 catalog_manager.cc:5484] T d404c85b11594e7eb1257fcf384beafa P a64152fd9595478ca767a9f248ef3b85 reported cstate change: config changed from index 6 to 7. New cstate: current_term: 1 leader_uuid: "a64152fd9595478ca767a9f248ef3b85" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:55:51.732831 27848 consensus_queue.cc:237] T d404c85b11594e7eb1257fcf384beafa P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: true } }
W20240502 01:55:51.736727 27784 consensus_peers.cc:485] T d404c85b11594e7eb1257fcf384beafa P a64152fd9595478ca767a9f248ef3b85 -> Peer c541378ca18944c49c37f49097bc7f81 (127.9.176.197:32773): Couldn't send request to peer c541378ca18944c49c37f49097bc7f81. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d404c85b11594e7eb1257fcf384beafa. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:51.737758 27715 raft_consensus.cc:1261] T d404c85b11594e7eb1257fcf384beafa P d50f07af87d74f039fb2b43cf166fa43 [term 1 FOLLOWER]: Refusing update from remote peer a64152fd9595478ca767a9f248ef3b85: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:51.738644 29539 consensus_queue.cc:1035] T d404c85b11594e7eb1257fcf384beafa P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:51.740727 28777 raft_consensus.cc:1261] T d404c85b11594e7eb1257fcf384beafa P 543169fd701141c181c55566d2cae00d [term 1 FOLLOWER]: Refusing update from remote peer a64152fd9595478ca767a9f248ef3b85: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:51.741791 29516 raft_consensus.cc:2932] T d404c85b11594e7eb1257fcf384beafa P a64152fd9595478ca767a9f248ef3b85 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER c541378ca18944c49c37f49097bc7f81 (127.9.176.197) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: true } } }
I20240502 01:55:51.742401 29525 consensus_queue.cc:1035] T d404c85b11594e7eb1257fcf384beafa P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:51.742470 27715 raft_consensus.cc:2932] T d404c85b11594e7eb1257fcf384beafa P d50f07af87d74f039fb2b43cf166fa43 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER c541378ca18944c49c37f49097bc7f81 (127.9.176.197) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: true } } }
I20240502 01:55:51.743693 28777 raft_consensus.cc:2932] T d404c85b11594e7eb1257fcf384beafa P 543169fd701141c181c55566d2cae00d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } }
I20240502 01:55:51.746063 27560 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet d404c85b11594e7eb1257fcf384beafa with cas_config_opid_index 7: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 01:55:51.748415 27572 catalog_manager.cc:5484] T d404c85b11594e7eb1257fcf384beafa P a64152fd9595478ca767a9f248ef3b85 reported cstate change: config changed from index 7 to 8, NON_VOTER c541378ca18944c49c37f49097bc7f81 (127.9.176.197) added. New cstate: current_term: 1 leader_uuid: "a64152fd9595478ca767a9f248ef3b85" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:55:51.756850 28777 raft_consensus.cc:2932] T d404c85b11594e7eb1257fcf384beafa P 543169fd701141c181c55566d2cae00d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER c541378ca18944c49c37f49097bc7f81 (127.9.176.197) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: true } } }
I20240502 01:55:52.150699 29582 ts_tablet_manager.cc:902] T 31de6c641c094cce894d8513b878350c P 543169fd701141c181c55566d2cae00d: Initiating tablet copy from peer a64152fd9595478ca767a9f248ef3b85 (127.9.176.194:38689)
I20240502 01:55:52.152392 29582 tablet_copy_client.cc:308] T 31de6c641c094cce894d8513b878350c P 543169fd701141c181c55566d2cae00d: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.194:38689
I20240502 01:55:52.153453 27869 tablet_copy_service.cc:140] P a64152fd9595478ca767a9f248ef3b85: Received BeginTabletCopySession request for tablet 31de6c641c094cce894d8513b878350c from peer 543169fd701141c181c55566d2cae00d ({username='slave'} at 127.9.176.196:38793)
I20240502 01:55:52.153657 27869 tablet_copy_service.cc:161] P a64152fd9595478ca767a9f248ef3b85: Beginning new tablet copy session on tablet 31de6c641c094cce894d8513b878350c from peer 543169fd701141c181c55566d2cae00d at {username='slave'} at 127.9.176.196:38793: session id = 543169fd701141c181c55566d2cae00d-31de6c641c094cce894d8513b878350c
I20240502 01:55:52.155915 27869 tablet_copy_source_session.cc:205] T 31de6c641c094cce894d8513b878350c P a64152fd9595478ca767a9f248ef3b85: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:52.157263 29582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31de6c641c094cce894d8513b878350c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:52.164523 29582 tablet_copy_client.cc:783] T 31de6c641c094cce894d8513b878350c P 543169fd701141c181c55566d2cae00d: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:52.164897 29582 tablet_copy_client.cc:647] T 31de6c641c094cce894d8513b878350c P 543169fd701141c181c55566d2cae00d: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:52.167117 29582 tablet_copy_client.cc:523] T 31de6c641c094cce894d8513b878350c P 543169fd701141c181c55566d2cae00d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:52.171159 29582 tablet_bootstrap.cc:492] T 31de6c641c094cce894d8513b878350c P 543169fd701141c181c55566d2cae00d: Bootstrap starting.
I20240502 01:55:52.206142 29582 tablet_bootstrap.cc:492] T 31de6c641c094cce894d8513b878350c P 543169fd701141c181c55566d2cae00d: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:52.206717 29582 tablet_bootstrap.cc:492] T 31de6c641c094cce894d8513b878350c P 543169fd701141c181c55566d2cae00d: Bootstrap complete.
I20240502 01:55:52.207038 29582 ts_tablet_manager.cc:1367] T 31de6c641c094cce894d8513b878350c P 543169fd701141c181c55566d2cae00d: Time spent bootstrapping tablet: real 0.036s	user 0.032s	sys 0.004s
I20240502 01:55:52.207787 29582 raft_consensus.cc:348] T 31de6c641c094cce894d8513b878350c P 543169fd701141c181c55566d2cae00d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: true } }
I20240502 01:55:52.208021 29582 raft_consensus.cc:729] T 31de6c641c094cce894d8513b878350c P 543169fd701141c181c55566d2cae00d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 543169fd701141c181c55566d2cae00d, State: Initialized, Role: LEARNER
I20240502 01:55:52.208216 29582 consensus_queue.cc:260] T 31de6c641c094cce894d8513b878350c P 543169fd701141c181c55566d2cae00d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: true } }
I20240502 01:55:52.208750 29582 ts_tablet_manager.cc:1398] T 31de6c641c094cce894d8513b878350c P 543169fd701141c181c55566d2cae00d: Time spent starting tablet: real 0.002s	user 0.004s	sys 0.000s
I20240502 01:55:52.209616 27869 tablet_copy_service.cc:342] P a64152fd9595478ca767a9f248ef3b85: Request end of tablet copy session 543169fd701141c181c55566d2cae00d-31de6c641c094cce894d8513b878350c received from {username='slave'} at 127.9.176.196:38793
I20240502 01:55:52.209767 27869 tablet_copy_service.cc:433] P a64152fd9595478ca767a9f248ef3b85: ending tablet copy session 543169fd701141c181c55566d2cae00d-31de6c641c094cce894d8513b878350c on tablet 31de6c641c094cce894d8513b878350c with peer 543169fd701141c181c55566d2cae00d
I20240502 01:55:52.234319 29603 ts_tablet_manager.cc:902] T d404c85b11594e7eb1257fcf384beafa P c541378ca18944c49c37f49097bc7f81: Initiating tablet copy from peer a64152fd9595478ca767a9f248ef3b85 (127.9.176.194:38689)
I20240502 01:55:52.235680 29603 tablet_copy_client.cc:308] T d404c85b11594e7eb1257fcf384beafa P c541378ca18944c49c37f49097bc7f81: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.194:38689
I20240502 01:55:52.236385 27869 tablet_copy_service.cc:140] P a64152fd9595478ca767a9f248ef3b85: Received BeginTabletCopySession request for tablet d404c85b11594e7eb1257fcf384beafa from peer c541378ca18944c49c37f49097bc7f81 ({username='slave'} at 127.9.176.197:47985)
I20240502 01:55:52.236585 27869 tablet_copy_service.cc:161] P a64152fd9595478ca767a9f248ef3b85: Beginning new tablet copy session on tablet d404c85b11594e7eb1257fcf384beafa from peer c541378ca18944c49c37f49097bc7f81 at {username='slave'} at 127.9.176.197:47985: session id = c541378ca18944c49c37f49097bc7f81-d404c85b11594e7eb1257fcf384beafa
I20240502 01:55:52.239491 27869 tablet_copy_source_session.cc:205] T d404c85b11594e7eb1257fcf384beafa P a64152fd9595478ca767a9f248ef3b85: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:52.240922 29603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d404c85b11594e7eb1257fcf384beafa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:52.245321 29603 tablet_copy_client.cc:783] T d404c85b11594e7eb1257fcf384beafa P c541378ca18944c49c37f49097bc7f81: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:52.245564 29603 tablet_copy_client.cc:647] T d404c85b11594e7eb1257fcf384beafa P c541378ca18944c49c37f49097bc7f81: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:52.247227 29603 tablet_copy_client.cc:523] T d404c85b11594e7eb1257fcf384beafa P c541378ca18944c49c37f49097bc7f81: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:52.249527 29603 tablet_bootstrap.cc:492] T d404c85b11594e7eb1257fcf384beafa P c541378ca18944c49c37f49097bc7f81: Bootstrap starting.
I20240502 01:55:52.258563 29582 ts_tablet_manager.cc:902] T f1597c56a1a74b37bb5e18e8c4662f46 P 543169fd701141c181c55566d2cae00d: Initiating tablet copy from peer a64152fd9595478ca767a9f248ef3b85 (127.9.176.194:38689)
I20240502 01:55:52.259869 29582 tablet_copy_client.cc:308] T f1597c56a1a74b37bb5e18e8c4662f46 P 543169fd701141c181c55566d2cae00d: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.194:38689
I20240502 01:55:52.260672 27869 tablet_copy_service.cc:140] P a64152fd9595478ca767a9f248ef3b85: Received BeginTabletCopySession request for tablet f1597c56a1a74b37bb5e18e8c4662f46 from peer 543169fd701141c181c55566d2cae00d ({username='slave'} at 127.9.176.196:38793)
I20240502 01:55:52.260845 27869 tablet_copy_service.cc:161] P a64152fd9595478ca767a9f248ef3b85: Beginning new tablet copy session on tablet f1597c56a1a74b37bb5e18e8c4662f46 from peer 543169fd701141c181c55566d2cae00d at {username='slave'} at 127.9.176.196:38793: session id = 543169fd701141c181c55566d2cae00d-f1597c56a1a74b37bb5e18e8c4662f46
I20240502 01:55:52.262811 27869 tablet_copy_source_session.cc:205] T f1597c56a1a74b37bb5e18e8c4662f46 P a64152fd9595478ca767a9f248ef3b85: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:52.264246 29582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1597c56a1a74b37bb5e18e8c4662f46. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:52.269562 29582 tablet_copy_client.cc:783] T f1597c56a1a74b37bb5e18e8c4662f46 P 543169fd701141c181c55566d2cae00d: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:52.269850 29582 tablet_copy_client.cc:647] T f1597c56a1a74b37bb5e18e8c4662f46 P 543169fd701141c181c55566d2cae00d: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:52.271728 29582 tablet_copy_client.cc:523] T f1597c56a1a74b37bb5e18e8c4662f46 P 543169fd701141c181c55566d2cae00d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:52.274391 29582 tablet_bootstrap.cc:492] T f1597c56a1a74b37bb5e18e8c4662f46 P 543169fd701141c181c55566d2cae00d: Bootstrap starting.
I20240502 01:55:52.285459 29603 tablet_bootstrap.cc:492] T d404c85b11594e7eb1257fcf384beafa P c541378ca18944c49c37f49097bc7f81: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:52.285921 29603 tablet_bootstrap.cc:492] T d404c85b11594e7eb1257fcf384beafa P c541378ca18944c49c37f49097bc7f81: Bootstrap complete.
I20240502 01:55:52.286202 29603 ts_tablet_manager.cc:1367] T d404c85b11594e7eb1257fcf384beafa P c541378ca18944c49c37f49097bc7f81: Time spent bootstrapping tablet: real 0.037s	user 0.028s	sys 0.007s
I20240502 01:55:52.286870 29603 raft_consensus.cc:348] T d404c85b11594e7eb1257fcf384beafa P c541378ca18944c49c37f49097bc7f81 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: true } }
I20240502 01:55:52.287106 29603 raft_consensus.cc:729] T d404c85b11594e7eb1257fcf384beafa P c541378ca18944c49c37f49097bc7f81 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c541378ca18944c49c37f49097bc7f81, State: Initialized, Role: LEARNER
I20240502 01:55:52.287330 29603 consensus_queue.cc:260] T d404c85b11594e7eb1257fcf384beafa P c541378ca18944c49c37f49097bc7f81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: true } }
I20240502 01:55:52.288331 29603 ts_tablet_manager.cc:1398] T d404c85b11594e7eb1257fcf384beafa P c541378ca18944c49c37f49097bc7f81: Time spent starting tablet: real 0.002s	user 0.004s	sys 0.000s
I20240502 01:55:52.289238 27869 tablet_copy_service.cc:342] P a64152fd9595478ca767a9f248ef3b85: Request end of tablet copy session c541378ca18944c49c37f49097bc7f81-d404c85b11594e7eb1257fcf384beafa received from {username='slave'} at 127.9.176.197:47985
I20240502 01:55:52.289426 27869 tablet_copy_service.cc:433] P a64152fd9595478ca767a9f248ef3b85: ending tablet copy session c541378ca18944c49c37f49097bc7f81-d404c85b11594e7eb1257fcf384beafa on tablet d404c85b11594e7eb1257fcf384beafa with peer c541378ca18944c49c37f49097bc7f81
I20240502 01:55:52.311388 29582 tablet_bootstrap.cc:492] T f1597c56a1a74b37bb5e18e8c4662f46 P 543169fd701141c181c55566d2cae00d: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:52.311894 29582 tablet_bootstrap.cc:492] T f1597c56a1a74b37bb5e18e8c4662f46 P 543169fd701141c181c55566d2cae00d: Bootstrap complete.
I20240502 01:55:52.312162 29582 ts_tablet_manager.cc:1367] T f1597c56a1a74b37bb5e18e8c4662f46 P 543169fd701141c181c55566d2cae00d: Time spent bootstrapping tablet: real 0.038s	user 0.032s	sys 0.008s
I20240502 01:55:52.312884 29582 raft_consensus.cc:348] T f1597c56a1a74b37bb5e18e8c4662f46 P 543169fd701141c181c55566d2cae00d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: true } }
I20240502 01:55:52.313110 29582 raft_consensus.cc:729] T f1597c56a1a74b37bb5e18e8c4662f46 P 543169fd701141c181c55566d2cae00d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 543169fd701141c181c55566d2cae00d, State: Initialized, Role: LEARNER
I20240502 01:55:52.313333 29582 consensus_queue.cc:260] T f1597c56a1a74b37bb5e18e8c4662f46 P 543169fd701141c181c55566d2cae00d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: true } }
I20240502 01:55:52.314002 29582 ts_tablet_manager.cc:1398] T f1597c56a1a74b37bb5e18e8c4662f46 P 543169fd701141c181c55566d2cae00d: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 01:55:52.314949 27869 tablet_copy_service.cc:342] P a64152fd9595478ca767a9f248ef3b85: Request end of tablet copy session 543169fd701141c181c55566d2cae00d-f1597c56a1a74b37bb5e18e8c4662f46 received from {username='slave'} at 127.9.176.196:38793
I20240502 01:55:52.315122 27869 tablet_copy_service.cc:433] P a64152fd9595478ca767a9f248ef3b85: ending tablet copy session 543169fd701141c181c55566d2cae00d-f1597c56a1a74b37bb5e18e8c4662f46 on tablet f1597c56a1a74b37bb5e18e8c4662f46 with peer 543169fd701141c181c55566d2cae00d
I20240502 01:55:52.746495 28777 raft_consensus.cc:1203] T 31de6c641c094cce894d8513b878350c P 543169fd701141c181c55566d2cae00d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8]   Dedup: 1.8->[]
I20240502 01:55:52.856060 28777 raft_consensus.cc:1203] T f1597c56a1a74b37bb5e18e8c4662f46 P 543169fd701141c181c55566d2cae00d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8]   Dedup: 1.8->[]
I20240502 01:55:52.859218 28645 raft_consensus.cc:1203] T d404c85b11594e7eb1257fcf384beafa P c541378ca18944c49c37f49097bc7f81 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8]   Dedup: 1.8->[]
I20240502 01:55:53.223557 29525 raft_consensus.cc:1050] T 31de6c641c094cce894d8513b878350c P a64152fd9595478ca767a9f248ef3b85: attempting to promote NON_VOTER 543169fd701141c181c55566d2cae00d to VOTER
I20240502 01:55:53.224345 29525 consensus_queue.cc:237] T 31de6c641c094cce894d8513b878350c P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } }
I20240502 01:55:53.226939 28777 raft_consensus.cc:1261] T 31de6c641c094cce894d8513b878350c P 543169fd701141c181c55566d2cae00d [term 1 LEARNER]: Refusing update from remote peer a64152fd9595478ca767a9f248ef3b85: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:53.227043 27981 raft_consensus.cc:1261] T 31de6c641c094cce894d8513b878350c P b550bed5a64f466c8ba5a1761f3bef39 [term 1 FOLLOWER]: Refusing update from remote peer a64152fd9595478ca767a9f248ef3b85: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:53.227082 28645 raft_consensus.cc:1261] T 31de6c641c094cce894d8513b878350c P c541378ca18944c49c37f49097bc7f81 [term 1 FOLLOWER]: Refusing update from remote peer a64152fd9595478ca767a9f248ef3b85: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:53.227630 29525 consensus_queue.cc:1035] T 31de6c641c094cce894d8513b878350c P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:53.228032 29680 consensus_queue.cc:1035] T 31de6c641c094cce894d8513b878350c P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:53.228382 29616 consensus_queue.cc:1035] T 31de6c641c094cce894d8513b878350c P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:53.231806 29616 raft_consensus.cc:2932] T 31de6c641c094cce894d8513b878350c P a64152fd9595478ca767a9f248ef3b85 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 543169fd701141c181c55566d2cae00d (127.9.176.196) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } }
I20240502 01:55:53.232528 28645 raft_consensus.cc:2932] T 31de6c641c094cce894d8513b878350c P c541378ca18944c49c37f49097bc7f81 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 543169fd701141c181c55566d2cae00d (127.9.176.196) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } }
I20240502 01:55:53.237432 27572 catalog_manager.cc:5484] T 31de6c641c094cce894d8513b878350c P c541378ca18944c49c37f49097bc7f81 reported cstate change: config changed from index 8 to 9, 543169fd701141c181c55566d2cae00d (127.9.176.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a64152fd9595478ca767a9f248ef3b85" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } }
I20240502 01:55:53.240316 28777 raft_consensus.cc:2932] T 31de6c641c094cce894d8513b878350c P 543169fd701141c181c55566d2cae00d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 543169fd701141c181c55566d2cae00d (127.9.176.196) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } }
I20240502 01:55:53.243680 27981 raft_consensus.cc:2932] T 31de6c641c094cce894d8513b878350c P b550bed5a64f466c8ba5a1761f3bef39 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 543169fd701141c181c55566d2cae00d (127.9.176.196) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } }
I20240502 01:55:53.323956 29680 raft_consensus.cc:1050] T f1597c56a1a74b37bb5e18e8c4662f46 P a64152fd9595478ca767a9f248ef3b85: attempting to promote NON_VOTER 543169fd701141c181c55566d2cae00d to VOTER
I20240502 01:55:53.324779 29680 consensus_queue.cc:237] T f1597c56a1a74b37bb5e18e8c4662f46 P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } }
I20240502 01:55:53.327308 28777 raft_consensus.cc:1261] T f1597c56a1a74b37bb5e18e8c4662f46 P 543169fd701141c181c55566d2cae00d [term 1 LEARNER]: Refusing update from remote peer a64152fd9595478ca767a9f248ef3b85: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:53.327488 27981 raft_consensus.cc:1261] T f1597c56a1a74b37bb5e18e8c4662f46 P b550bed5a64f466c8ba5a1761f3bef39 [term 1 FOLLOWER]: Refusing update from remote peer a64152fd9595478ca767a9f248ef3b85: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:53.327840 28645 raft_consensus.cc:1261] T f1597c56a1a74b37bb5e18e8c4662f46 P c541378ca18944c49c37f49097bc7f81 [term 1 FOLLOWER]: Refusing update from remote peer a64152fd9595478ca767a9f248ef3b85: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:53.327926 29680 consensus_queue.cc:1035] T f1597c56a1a74b37bb5e18e8c4662f46 P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:53.328394 29525 consensus_queue.cc:1035] T f1597c56a1a74b37bb5e18e8c4662f46 P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:53.328714 29616 consensus_queue.cc:1035] T f1597c56a1a74b37bb5e18e8c4662f46 P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:53.332557 29680 raft_consensus.cc:2932] T f1597c56a1a74b37bb5e18e8c4662f46 P a64152fd9595478ca767a9f248ef3b85 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 543169fd701141c181c55566d2cae00d (127.9.176.196) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } }
I20240502 01:55:53.333371 28777 raft_consensus.cc:2932] T f1597c56a1a74b37bb5e18e8c4662f46 P 543169fd701141c181c55566d2cae00d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 543169fd701141c181c55566d2cae00d (127.9.176.196) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } }
I20240502 01:55:53.333348 28645 raft_consensus.cc:2932] T f1597c56a1a74b37bb5e18e8c4662f46 P c541378ca18944c49c37f49097bc7f81 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 543169fd701141c181c55566d2cae00d (127.9.176.196) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } }
I20240502 01:55:53.336702 27981 raft_consensus.cc:2932] T f1597c56a1a74b37bb5e18e8c4662f46 P b550bed5a64f466c8ba5a1761f3bef39 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 543169fd701141c181c55566d2cae00d (127.9.176.196) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } }
I20240502 01:55:53.339385 27570 catalog_manager.cc:5484] T f1597c56a1a74b37bb5e18e8c4662f46 P a64152fd9595478ca767a9f248ef3b85 reported cstate change: config changed from index 8 to 9, 543169fd701141c181c55566d2cae00d (127.9.176.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a64152fd9595478ca767a9f248ef3b85" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:55:53.399701 27848 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f1597c56a1a74b37bb5e18e8c4662f46"
dest_uuid: "a64152fd9595478ca767a9f248ef3b85"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:53050
I20240502 01:55:53.400017 27848 raft_consensus.cc:595] T f1597c56a1a74b37bb5e18e8c4662f46 P a64152fd9595478ca767a9f248ef3b85 [term 1 LEADER]: Received request to transfer leadership
I20240502 01:55:53.420804 27848 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "31de6c641c094cce894d8513b878350c"
dest_uuid: "a64152fd9595478ca767a9f248ef3b85"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:53060
I20240502 01:55:53.421027 27848 raft_consensus.cc:595] T 31de6c641c094cce894d8513b878350c P a64152fd9595478ca767a9f248ef3b85 [term 1 LEADER]: Received request to transfer leadership
I20240502 01:55:53.444518 29525 raft_consensus.cc:1050] T d404c85b11594e7eb1257fcf384beafa P a64152fd9595478ca767a9f248ef3b85: attempting to promote NON_VOTER c541378ca18944c49c37f49097bc7f81 to VOTER
I20240502 01:55:53.445304 29525 consensus_queue.cc:237] T d404c85b11594e7eb1257fcf384beafa P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } }
I20240502 01:55:53.447729 28645 raft_consensus.cc:1261] T d404c85b11594e7eb1257fcf384beafa P c541378ca18944c49c37f49097bc7f81 [term 1 LEARNER]: Refusing update from remote peer a64152fd9595478ca767a9f248ef3b85: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:53.447731 27715 raft_consensus.cc:1261] T d404c85b11594e7eb1257fcf384beafa P d50f07af87d74f039fb2b43cf166fa43 [term 1 FOLLOWER]: Refusing update from remote peer a64152fd9595478ca767a9f248ef3b85: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:53.447732 28777 raft_consensus.cc:1261] T d404c85b11594e7eb1257fcf384beafa P 543169fd701141c181c55566d2cae00d [term 1 FOLLOWER]: Refusing update from remote peer a64152fd9595478ca767a9f248ef3b85: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:53.448340 29615 consensus_queue.cc:1035] T d404c85b11594e7eb1257fcf384beafa P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:53.448753 29616 consensus_queue.cc:1035] T d404c85b11594e7eb1257fcf384beafa P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:53.449100 29525 consensus_queue.cc:1035] T d404c85b11594e7eb1257fcf384beafa P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:53.452626 29615 raft_consensus.cc:2932] T d404c85b11594e7eb1257fcf384beafa P a64152fd9595478ca767a9f248ef3b85 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, c541378ca18944c49c37f49097bc7f81 (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } }
I20240502 01:55:53.453348 28777 raft_consensus.cc:2932] T d404c85b11594e7eb1257fcf384beafa P 543169fd701141c181c55566d2cae00d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, c541378ca18944c49c37f49097bc7f81 (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } }
I20240502 01:55:53.454041 28645 raft_consensus.cc:2932] T d404c85b11594e7eb1257fcf384beafa P c541378ca18944c49c37f49097bc7f81 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, c541378ca18944c49c37f49097bc7f81 (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } }
I20240502 01:55:53.459065 27572 catalog_manager.cc:5484] T d404c85b11594e7eb1257fcf384beafa P c541378ca18944c49c37f49097bc7f81 reported cstate change: config changed from index 8 to 9, c541378ca18944c49c37f49097bc7f81 (127.9.176.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a64152fd9595478ca767a9f248ef3b85" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } }
I20240502 01:55:53.459019 27715 raft_consensus.cc:2932] T d404c85b11594e7eb1257fcf384beafa P d50f07af87d74f039fb2b43cf166fa43 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, c541378ca18944c49c37f49097bc7f81 (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } }
I20240502 01:55:53.633316 29525 raft_consensus.cc:979] T 31de6c641c094cce894d8513b878350c P a64152fd9595478ca767a9f248ef3b85: : Instructing follower 543169fd701141c181c55566d2cae00d to start an election
I20240502 01:55:53.633515 29615 raft_consensus.cc:1067] T 31de6c641c094cce894d8513b878350c P a64152fd9595478ca767a9f248ef3b85 [term 1 LEADER]: Signalling peer 543169fd701141c181c55566d2cae00d to start an election
I20240502 01:55:53.634426 28777 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "31de6c641c094cce894d8513b878350c"
dest_uuid: "543169fd701141c181c55566d2cae00d"
 from {username='slave'} at 127.9.176.194:36721
I20240502 01:55:53.634711 28777 raft_consensus.cc:482] T 31de6c641c094cce894d8513b878350c P 543169fd701141c181c55566d2cae00d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:55:53.634887 28777 raft_consensus.cc:3037] T 31de6c641c094cce894d8513b878350c P 543169fd701141c181c55566d2cae00d [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:55:53.637858 28777 raft_consensus.cc:504] T 31de6c641c094cce894d8513b878350c P 543169fd701141c181c55566d2cae00d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } }
I20240502 01:55:53.639683 28777 leader_election.cc:288] T 31de6c641c094cce894d8513b878350c P 543169fd701141c181c55566d2cae00d [CANDIDATE]: Term 2 election: Requested vote from peers a64152fd9595478ca767a9f248ef3b85 (127.9.176.194:38689), b550bed5a64f466c8ba5a1761f3bef39 (127.9.176.195:40013), c541378ca18944c49c37f49097bc7f81 (127.9.176.197:32773)
I20240502 01:55:53.640152 27848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31de6c641c094cce894d8513b878350c" candidate_uuid: "543169fd701141c181c55566d2cae00d" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "a64152fd9595478ca767a9f248ef3b85"
I20240502 01:55:53.640457 27848 raft_consensus.cc:3032] T 31de6c641c094cce894d8513b878350c P a64152fd9595478ca767a9f248ef3b85 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:55:53.640596 27848 raft_consensus.cc:729] T 31de6c641c094cce894d8513b878350c P a64152fd9595478ca767a9f248ef3b85 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a64152fd9595478ca767a9f248ef3b85, State: Running, Role: LEADER
I20240502 01:55:53.641134 27848 consensus_queue.cc:260] T 31de6c641c094cce894d8513b878350c P a64152fd9595478ca767a9f248ef3b85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } }
I20240502 01:55:53.641669 27848 raft_consensus.cc:3037] T 31de6c641c094cce894d8513b878350c P a64152fd9595478ca767a9f248ef3b85 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:55:53.650588 27848 raft_consensus.cc:2454] T 31de6c641c094cce894d8513b878350c P a64152fd9595478ca767a9f248ef3b85 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 543169fd701141c181c55566d2cae00d in term 2.
I20240502 01:55:53.657001 27848 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d404c85b11594e7eb1257fcf384beafa"
dest_uuid: "a64152fd9595478ca767a9f248ef3b85"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:53084
I20240502 01:55:53.657258 27848 raft_consensus.cc:595] T d404c85b11594e7eb1257fcf384beafa P a64152fd9595478ca767a9f248ef3b85 [term 1 LEADER]: Received request to transfer leadership
I20240502 01:55:53.657752 27981 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31de6c641c094cce894d8513b878350c" candidate_uuid: "543169fd701141c181c55566d2cae00d" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "b550bed5a64f466c8ba5a1761f3bef39"
I20240502 01:55:53.658020 27981 raft_consensus.cc:3037] T 31de6c641c094cce894d8513b878350c P b550bed5a64f466c8ba5a1761f3bef39 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:55:53.661168 27981 raft_consensus.cc:2454] T 31de6c641c094cce894d8513b878350c P b550bed5a64f466c8ba5a1761f3bef39 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 543169fd701141c181c55566d2cae00d in term 2.
I20240502 01:55:53.662151 28713 leader_election.cc:302] T 31de6c641c094cce894d8513b878350c P 543169fd701141c181c55566d2cae00d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 543169fd701141c181c55566d2cae00d, a64152fd9595478ca767a9f248ef3b85, b550bed5a64f466c8ba5a1761f3bef39; no voters: 
I20240502 01:55:53.662680 29561 raft_consensus.cc:2781] T 31de6c641c094cce894d8513b878350c P 543169fd701141c181c55566d2cae00d [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:55:53.663368 29561 raft_consensus.cc:686] T 31de6c641c094cce894d8513b878350c P 543169fd701141c181c55566d2cae00d [term 2 LEADER]: Becoming Leader. State: Replica: 543169fd701141c181c55566d2cae00d, State: Running, Role: LEADER
I20240502 01:55:53.664232 29561 consensus_queue.cc:237] T 31de6c641c094cce894d8513b878350c P 543169fd701141c181c55566d2cae00d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } }
I20240502 01:55:53.666922 28645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31de6c641c094cce894d8513b878350c" candidate_uuid: "543169fd701141c181c55566d2cae00d" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "c541378ca18944c49c37f49097bc7f81"
I20240502 01:55:53.667265 28645 raft_consensus.cc:3037] T 31de6c641c094cce894d8513b878350c P c541378ca18944c49c37f49097bc7f81 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:55:53.670666 27566 catalog_manager.cc:5484] T 31de6c641c094cce894d8513b878350c P 543169fd701141c181c55566d2cae00d reported cstate change: term changed from 1 to 2, leader changed from a64152fd9595478ca767a9f248ef3b85 (127.9.176.194) to 543169fd701141c181c55566d2cae00d (127.9.176.196). New cstate: current_term: 2 leader_uuid: "543169fd701141c181c55566d2cae00d" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:55:53.672238 28645 raft_consensus.cc:2454] T 31de6c641c094cce894d8513b878350c P c541378ca18944c49c37f49097bc7f81 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 543169fd701141c181c55566d2cae00d in term 2.
I20240502 01:55:53.689836 27848 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f1597c56a1a74b37bb5e18e8c4662f46"
dest_uuid: "a64152fd9595478ca767a9f248ef3b85"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:53098
I20240502 01:55:53.690083 27848 raft_consensus.cc:595] T f1597c56a1a74b37bb5e18e8c4662f46 P a64152fd9595478ca767a9f248ef3b85 [term 1 LEADER]: Received request to transfer leadership
I20240502 01:55:53.771435 29525 raft_consensus.cc:979] T f1597c56a1a74b37bb5e18e8c4662f46 P a64152fd9595478ca767a9f248ef3b85: : Instructing follower c541378ca18944c49c37f49097bc7f81 to start an election
I20240502 01:55:53.771639 29615 raft_consensus.cc:1067] T f1597c56a1a74b37bb5e18e8c4662f46 P a64152fd9595478ca767a9f248ef3b85 [term 1 LEADER]: Signalling peer c541378ca18944c49c37f49097bc7f81 to start an election
I20240502 01:55:53.772519 28645 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f1597c56a1a74b37bb5e18e8c4662f46"
dest_uuid: "c541378ca18944c49c37f49097bc7f81"
 from {username='slave'} at 127.9.176.194:59883
I20240502 01:55:53.772842 28645 raft_consensus.cc:482] T f1597c56a1a74b37bb5e18e8c4662f46 P c541378ca18944c49c37f49097bc7f81 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:55:53.772972 28645 raft_consensus.cc:3037] T f1597c56a1a74b37bb5e18e8c4662f46 P c541378ca18944c49c37f49097bc7f81 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:55:53.775241 28645 raft_consensus.cc:504] T f1597c56a1a74b37bb5e18e8c4662f46 P c541378ca18944c49c37f49097bc7f81 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } }
I20240502 01:55:53.777024 28645 leader_election.cc:288] T f1597c56a1a74b37bb5e18e8c4662f46 P c541378ca18944c49c37f49097bc7f81 [CANDIDATE]: Term 2 election: Requested vote from peers a64152fd9595478ca767a9f248ef3b85 (127.9.176.194:38689), b550bed5a64f466c8ba5a1761f3bef39 (127.9.176.195:40013), 543169fd701141c181c55566d2cae00d (127.9.176.196:38773)
I20240502 01:55:53.777940 27848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1597c56a1a74b37bb5e18e8c4662f46" candidate_uuid: "c541378ca18944c49c37f49097bc7f81" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "a64152fd9595478ca767a9f248ef3b85"
I20240502 01:55:53.778226 27848 raft_consensus.cc:3032] T f1597c56a1a74b37bb5e18e8c4662f46 P a64152fd9595478ca767a9f248ef3b85 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:55:53.778337 27848 raft_consensus.cc:729] T f1597c56a1a74b37bb5e18e8c4662f46 P a64152fd9595478ca767a9f248ef3b85 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a64152fd9595478ca767a9f248ef3b85, State: Running, Role: LEADER
I20240502 01:55:53.778707 27848 consensus_queue.cc:260] T f1597c56a1a74b37bb5e18e8c4662f46 P a64152fd9595478ca767a9f248ef3b85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } }
I20240502 01:55:53.779253 27848 raft_consensus.cc:3037] T f1597c56a1a74b37bb5e18e8c4662f46 P a64152fd9595478ca767a9f248ef3b85 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:55:53.781803 27848 raft_consensus.cc:2454] T f1597c56a1a74b37bb5e18e8c4662f46 P a64152fd9595478ca767a9f248ef3b85 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c541378ca18944c49c37f49097bc7f81 in term 2.
I20240502 01:55:53.786581 27981 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1597c56a1a74b37bb5e18e8c4662f46" candidate_uuid: "c541378ca18944c49c37f49097bc7f81" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "b550bed5a64f466c8ba5a1761f3bef39"
I20240502 01:55:53.786851 27981 raft_consensus.cc:3037] T f1597c56a1a74b37bb5e18e8c4662f46 P b550bed5a64f466c8ba5a1761f3bef39 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:55:53.786779 28777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1597c56a1a74b37bb5e18e8c4662f46" candidate_uuid: "c541378ca18944c49c37f49097bc7f81" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "543169fd701141c181c55566d2cae00d"
I20240502 01:55:53.787107 28777 raft_consensus.cc:3037] T f1597c56a1a74b37bb5e18e8c4662f46 P 543169fd701141c181c55566d2cae00d [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:55:53.788949 27981 raft_consensus.cc:2454] T f1597c56a1a74b37bb5e18e8c4662f46 P b550bed5a64f466c8ba5a1761f3bef39 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c541378ca18944c49c37f49097bc7f81 in term 2.
I20240502 01:55:53.789320 28777 raft_consensus.cc:2454] T f1597c56a1a74b37bb5e18e8c4662f46 P 543169fd701141c181c55566d2cae00d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c541378ca18944c49c37f49097bc7f81 in term 2.
I20240502 01:55:53.789537 28580 leader_election.cc:302] T f1597c56a1a74b37bb5e18e8c4662f46 P c541378ca18944c49c37f49097bc7f81 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: a64152fd9595478ca767a9f248ef3b85, b550bed5a64f466c8ba5a1761f3bef39, c541378ca18944c49c37f49097bc7f81; no voters: 
I20240502 01:55:53.790046 29648 raft_consensus.cc:2781] T f1597c56a1a74b37bb5e18e8c4662f46 P c541378ca18944c49c37f49097bc7f81 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:55:53.790616 29648 raft_consensus.cc:686] T f1597c56a1a74b37bb5e18e8c4662f46 P c541378ca18944c49c37f49097bc7f81 [term 2 LEADER]: Becoming Leader. State: Replica: c541378ca18944c49c37f49097bc7f81, State: Running, Role: LEADER
I20240502 01:55:53.791219 29648 consensus_queue.cc:237] T f1597c56a1a74b37bb5e18e8c4662f46 P c541378ca18944c49c37f49097bc7f81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } }
I20240502 01:55:53.795179 27566 catalog_manager.cc:5484] T f1597c56a1a74b37bb5e18e8c4662f46 P c541378ca18944c49c37f49097bc7f81 reported cstate change: term changed from 1 to 2, leader changed from a64152fd9595478ca767a9f248ef3b85 (127.9.176.194) to c541378ca18944c49c37f49097bc7f81 (127.9.176.197). New cstate: current_term: 2 leader_uuid: "c541378ca18944c49c37f49097bc7f81" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:55:53.920936 27848 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d404c85b11594e7eb1257fcf384beafa"
dest_uuid: "a64152fd9595478ca767a9f248ef3b85"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:53128
I20240502 01:55:53.921169 27848 raft_consensus.cc:595] T d404c85b11594e7eb1257fcf384beafa P a64152fd9595478ca767a9f248ef3b85 [term 1 LEADER]: Received request to transfer leadership
I20240502 01:55:53.933342 29615 raft_consensus.cc:979] T d404c85b11594e7eb1257fcf384beafa P a64152fd9595478ca767a9f248ef3b85: : Instructing follower c541378ca18944c49c37f49097bc7f81 to start an election
I20240502 01:55:53.933554 29615 raft_consensus.cc:1067] T d404c85b11594e7eb1257fcf384beafa P a64152fd9595478ca767a9f248ef3b85 [term 1 LEADER]: Signalling peer c541378ca18944c49c37f49097bc7f81 to start an election
I20240502 01:55:53.934242 28645 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d404c85b11594e7eb1257fcf384beafa"
dest_uuid: "c541378ca18944c49c37f49097bc7f81"
 from {username='slave'} at 127.9.176.194:59883
I20240502 01:55:53.934471 28645 raft_consensus.cc:482] T d404c85b11594e7eb1257fcf384beafa P c541378ca18944c49c37f49097bc7f81 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:55:53.934661 28645 raft_consensus.cc:3037] T d404c85b11594e7eb1257fcf384beafa P c541378ca18944c49c37f49097bc7f81 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:55:53.938352 28645 raft_consensus.cc:504] T d404c85b11594e7eb1257fcf384beafa P c541378ca18944c49c37f49097bc7f81 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } }
I20240502 01:55:53.939909 27848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d404c85b11594e7eb1257fcf384beafa" candidate_uuid: "c541378ca18944c49c37f49097bc7f81" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "a64152fd9595478ca767a9f248ef3b85"
I20240502 01:55:53.940164 27848 raft_consensus.cc:3032] T d404c85b11594e7eb1257fcf384beafa P a64152fd9595478ca767a9f248ef3b85 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:55:53.940277 27848 raft_consensus.cc:729] T d404c85b11594e7eb1257fcf384beafa P a64152fd9595478ca767a9f248ef3b85 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a64152fd9595478ca767a9f248ef3b85, State: Running, Role: LEADER
I20240502 01:55:53.940276 28777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d404c85b11594e7eb1257fcf384beafa" candidate_uuid: "c541378ca18944c49c37f49097bc7f81" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "543169fd701141c181c55566d2cae00d"
I20240502 01:55:53.940534 28777 raft_consensus.cc:3037] T d404c85b11594e7eb1257fcf384beafa P 543169fd701141c181c55566d2cae00d [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:55:53.940622 27848 consensus_queue.cc:260] T d404c85b11594e7eb1257fcf384beafa P a64152fd9595478ca767a9f248ef3b85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } }
I20240502 01:55:53.941056 27848 raft_consensus.cc:3037] T d404c85b11594e7eb1257fcf384beafa P a64152fd9595478ca767a9f248ef3b85 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:55:53.943202 27848 raft_consensus.cc:2454] T d404c85b11594e7eb1257fcf384beafa P a64152fd9595478ca767a9f248ef3b85 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c541378ca18944c49c37f49097bc7f81 in term 2.
I20240502 01:55:53.943353 28777 raft_consensus.cc:2454] T d404c85b11594e7eb1257fcf384beafa P 543169fd701141c181c55566d2cae00d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c541378ca18944c49c37f49097bc7f81 in term 2.
I20240502 01:55:53.939431 28645 leader_election.cc:288] T d404c85b11594e7eb1257fcf384beafa P c541378ca18944c49c37f49097bc7f81 [CANDIDATE]: Term 2 election: Requested vote from peers d50f07af87d74f039fb2b43cf166fa43 (127.9.176.193:34203), a64152fd9595478ca767a9f248ef3b85 (127.9.176.194:38689), 543169fd701141c181c55566d2cae00d (127.9.176.196:38773)
I20240502 01:55:53.943874 28578 leader_election.cc:302] T d404c85b11594e7eb1257fcf384beafa P c541378ca18944c49c37f49097bc7f81 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 543169fd701141c181c55566d2cae00d, a64152fd9595478ca767a9f248ef3b85, c541378ca18944c49c37f49097bc7f81; no voters: 
I20240502 01:55:53.946736 29648 raft_consensus.cc:2781] T d404c85b11594e7eb1257fcf384beafa P c541378ca18944c49c37f49097bc7f81 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:55:53.946918 29648 raft_consensus.cc:686] T d404c85b11594e7eb1257fcf384beafa P c541378ca18944c49c37f49097bc7f81 [term 2 LEADER]: Becoming Leader. State: Replica: c541378ca18944c49c37f49097bc7f81, State: Running, Role: LEADER
I20240502 01:55:53.947245 29648 consensus_queue.cc:237] T d404c85b11594e7eb1257fcf384beafa P c541378ca18944c49c37f49097bc7f81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } }
I20240502 01:55:53.950861 27566 catalog_manager.cc:5484] T d404c85b11594e7eb1257fcf384beafa P c541378ca18944c49c37f49097bc7f81 reported cstate change: term changed from 1 to 2, leader changed from a64152fd9595478ca767a9f248ef3b85 (127.9.176.194) to c541378ca18944c49c37f49097bc7f81 (127.9.176.197). New cstate: current_term: 2 leader_uuid: "c541378ca18944c49c37f49097bc7f81" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:55:53.959215 27715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d404c85b11594e7eb1257fcf384beafa" candidate_uuid: "c541378ca18944c49c37f49097bc7f81" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "d50f07af87d74f039fb2b43cf166fa43"
I20240502 01:55:53.959442 27715 raft_consensus.cc:3037] T d404c85b11594e7eb1257fcf384beafa P d50f07af87d74f039fb2b43cf166fa43 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:55:53.961669 27715 raft_consensus.cc:2454] T d404c85b11594e7eb1257fcf384beafa P d50f07af87d74f039fb2b43cf166fa43 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c541378ca18944c49c37f49097bc7f81 in term 2.
I20240502 01:55:54.082085 27848 raft_consensus.cc:1261] T 31de6c641c094cce894d8513b878350c P a64152fd9595478ca767a9f248ef3b85 [term 2 FOLLOWER]: Refusing update from remote peer 543169fd701141c181c55566d2cae00d: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 01:55:54.083007 29561 consensus_queue.cc:1035] T 31de6c641c094cce894d8513b878350c P 543169fd701141c181c55566d2cae00d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:54.086901 27981 raft_consensus.cc:1261] T 31de6c641c094cce894d8513b878350c P b550bed5a64f466c8ba5a1761f3bef39 [term 2 FOLLOWER]: Refusing update from remote peer 543169fd701141c181c55566d2cae00d: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 01:55:54.087793 29561 consensus_queue.cc:1035] T 31de6c641c094cce894d8513b878350c P 543169fd701141c181c55566d2cae00d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:54.093566 28645 raft_consensus.cc:1261] T 31de6c641c094cce894d8513b878350c P c541378ca18944c49c37f49097bc7f81 [term 2 FOLLOWER]: Refusing update from remote peer 543169fd701141c181c55566d2cae00d: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 01:55:54.094360 29561 consensus_queue.cc:1035] T 31de6c641c094cce894d8513b878350c P 543169fd701141c181c55566d2cae00d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:54.303692 27981 raft_consensus.cc:1261] T f1597c56a1a74b37bb5e18e8c4662f46 P b550bed5a64f466c8ba5a1761f3bef39 [term 2 FOLLOWER]: Refusing update from remote peer c541378ca18944c49c37f49097bc7f81: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 01:55:54.304461 29648 consensus_queue.cc:1035] T f1597c56a1a74b37bb5e18e8c4662f46 P c541378ca18944c49c37f49097bc7f81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:54.350481 27849 raft_consensus.cc:1261] T d404c85b11594e7eb1257fcf384beafa P a64152fd9595478ca767a9f248ef3b85 [term 2 FOLLOWER]: Refusing update from remote peer c541378ca18944c49c37f49097bc7f81: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 01:55:54.351157 29648 consensus_queue.cc:1035] T d404c85b11594e7eb1257fcf384beafa P c541378ca18944c49c37f49097bc7f81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:54.354262 28778 raft_consensus.cc:1261] T f1597c56a1a74b37bb5e18e8c4662f46 P 543169fd701141c181c55566d2cae00d [term 2 FOLLOWER]: Refusing update from remote peer c541378ca18944c49c37f49097bc7f81: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 01:55:54.354869 29648 consensus_queue.cc:1035] T f1597c56a1a74b37bb5e18e8c4662f46 P c541378ca18944c49c37f49097bc7f81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:54.359426 27849 raft_consensus.cc:1261] T f1597c56a1a74b37bb5e18e8c4662f46 P a64152fd9595478ca767a9f248ef3b85 [term 2 FOLLOWER]: Refusing update from remote peer c541378ca18944c49c37f49097bc7f81: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 01:55:54.360177 29799 consensus_queue.cc:1035] T f1597c56a1a74b37bb5e18e8c4662f46 P c541378ca18944c49c37f49097bc7f81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:54.375883 28645 consensus_queue.cc:237] T f1597c56a1a74b37bb5e18e8c4662f46 P c541378ca18944c49c37f49097bc7f81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } }
I20240502 01:55:54.378072 28778 raft_consensus.cc:1261] T f1597c56a1a74b37bb5e18e8c4662f46 P 543169fd701141c181c55566d2cae00d [term 2 FOLLOWER]: Refusing update from remote peer c541378ca18944c49c37f49097bc7f81: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240502 01:55:54.378362 27981 raft_consensus.cc:1261] T f1597c56a1a74b37bb5e18e8c4662f46 P b550bed5a64f466c8ba5a1761f3bef39 [term 2 FOLLOWER]: Refusing update from remote peer c541378ca18944c49c37f49097bc7f81: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240502 01:55:54.378888 29799 consensus_queue.cc:1035] T f1597c56a1a74b37bb5e18e8c4662f46 P c541378ca18944c49c37f49097bc7f81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 01:55:54.379148 29648 consensus_queue.cc:1035] T f1597c56a1a74b37bb5e18e8c4662f46 P c541378ca18944c49c37f49097bc7f81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 01:55:54.381573 29755 raft_consensus.cc:2932] T f1597c56a1a74b37bb5e18e8c4662f46 P c541378ca18944c49c37f49097bc7f81 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER a64152fd9595478ca767a9f248ef3b85 (127.9.176.194) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } }
I20240502 01:55:54.382288 27981 raft_consensus.cc:2932] T f1597c56a1a74b37bb5e18e8c4662f46 P b550bed5a64f466c8ba5a1761f3bef39 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER a64152fd9595478ca767a9f248ef3b85 (127.9.176.194) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } }
I20240502 01:55:54.384105 28778 raft_consensus.cc:2932] T f1597c56a1a74b37bb5e18e8c4662f46 P 543169fd701141c181c55566d2cae00d [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER a64152fd9595478ca767a9f248ef3b85 (127.9.176.194) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } }
I20240502 01:55:54.387635 27572 catalog_manager.cc:5484] T f1597c56a1a74b37bb5e18e8c4662f46 P b550bed5a64f466c8ba5a1761f3bef39 reported cstate change: config changed from index 9 to 11, VOTER a64152fd9595478ca767a9f248ef3b85 (127.9.176.194) evicted. New cstate: current_term: 2 leader_uuid: "c541378ca18944c49c37f49097bc7f81" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } }
I20240502 01:55:54.388527 27715 raft_consensus.cc:1261] T d404c85b11594e7eb1257fcf384beafa P d50f07af87d74f039fb2b43cf166fa43 [term 2 FOLLOWER]: Refusing update from remote peer c541378ca18944c49c37f49097bc7f81: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 01:55:54.390344 29648 consensus_queue.cc:1035] T d404c85b11594e7eb1257fcf384beafa P c541378ca18944c49c37f49097bc7f81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:54.391671 27560 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f1597c56a1a74b37bb5e18e8c4662f46 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:55:54.398303 28778 raft_consensus.cc:1261] T d404c85b11594e7eb1257fcf384beafa P 543169fd701141c181c55566d2cae00d [term 2 FOLLOWER]: Refusing update from remote peer c541378ca18944c49c37f49097bc7f81: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 01:55:54.399019 29755 consensus_queue.cc:1035] T d404c85b11594e7eb1257fcf384beafa P c541378ca18944c49c37f49097bc7f81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:54.404397 27827 tablet_service.cc:1514] Processing DeleteTablet for tablet f1597c56a1a74b37bb5e18e8c4662f46 with delete_type TABLET_DATA_TOMBSTONED (TS a64152fd9595478ca767a9f248ef3b85 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:55806
I20240502 01:55:54.408231 29805 tablet_replica.cc:330] T f1597c56a1a74b37bb5e18e8c4662f46 P a64152fd9595478ca767a9f248ef3b85: stopping tablet replica
I20240502 01:55:54.408833 29805 raft_consensus.cc:2229] T f1597c56a1a74b37bb5e18e8c4662f46 P a64152fd9595478ca767a9f248ef3b85 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:54.409130 29805 raft_consensus.cc:2258] T f1597c56a1a74b37bb5e18e8c4662f46 P a64152fd9595478ca767a9f248ef3b85 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:54.410634 29805 ts_tablet_manager.cc:1875] T f1597c56a1a74b37bb5e18e8c4662f46 P a64152fd9595478ca767a9f248ef3b85: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:54.416249 29805 ts_tablet_manager.cc:1888] T f1597c56a1a74b37bb5e18e8c4662f46 P a64152fd9595478ca767a9f248ef3b85: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.10
I20240502 01:55:54.416435 29805 log.cc:1196] T f1597c56a1a74b37bb5e18e8c4662f46 P a64152fd9595478ca767a9f248ef3b85: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/wal/wals/f1597c56a1a74b37bb5e18e8c4662f46
I20240502 01:55:54.417493 27560 catalog_manager.cc:4830] TS a64152fd9595478ca767a9f248ef3b85 (127.9.176.194:38689): tablet f1597c56a1a74b37bb5e18e8c4662f46 (table rebalance_test_table_0 [id=8ef52b859d42498b92b6b2ec0805663b]) successfully deleted
I20240502 01:55:54.661252 28778 consensus_queue.cc:237] T 31de6c641c094cce894d8513b878350c P 543169fd701141c181c55566d2cae00d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } }
I20240502 01:55:54.663538 28645 raft_consensus.cc:1261] T 31de6c641c094cce894d8513b878350c P c541378ca18944c49c37f49097bc7f81 [term 2 FOLLOWER]: Refusing update from remote peer 543169fd701141c181c55566d2cae00d: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240502 01:55:54.663650 27981 raft_consensus.cc:1261] T 31de6c641c094cce894d8513b878350c P b550bed5a64f466c8ba5a1761f3bef39 [term 2 FOLLOWER]: Refusing update from remote peer 543169fd701141c181c55566d2cae00d: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240502 01:55:54.664207 27849 consensus_queue.cc:237] T 63136d6964084523bb950cc6568a273b P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } }
I20240502 01:55:54.664294 29825 consensus_queue.cc:1035] T 31de6c641c094cce894d8513b878350c P 543169fd701141c181c55566d2cae00d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 01:55:54.664774 29823 consensus_queue.cc:1035] T 31de6c641c094cce894d8513b878350c P 543169fd701141c181c55566d2cae00d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 01:55:54.667930 29737 raft_consensus.cc:2932] T 31de6c641c094cce894d8513b878350c P 543169fd701141c181c55566d2cae00d [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER a64152fd9595478ca767a9f248ef3b85 (127.9.176.194) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } }
I20240502 01:55:54.669066 27981 raft_consensus.cc:2932] T 31de6c641c094cce894d8513b878350c P b550bed5a64f466c8ba5a1761f3bef39 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER a64152fd9595478ca767a9f248ef3b85 (127.9.176.194) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } }
I20240502 01:55:54.674376 28645 raft_consensus.cc:2932] T 31de6c641c094cce894d8513b878350c P c541378ca18944c49c37f49097bc7f81 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER a64152fd9595478ca767a9f248ef3b85 (127.9.176.194) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } }
I20240502 01:55:54.675856 27715 raft_consensus.cc:1261] T 63136d6964084523bb950cc6568a273b P d50f07af87d74f039fb2b43cf166fa43 [term 1 FOLLOWER]: Refusing update from remote peer a64152fd9595478ca767a9f248ef3b85: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:54.676513 29615 consensus_queue.cc:1035] T 63136d6964084523bb950cc6568a273b P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:54.676712 27572 catalog_manager.cc:5484] T 31de6c641c094cce894d8513b878350c P b550bed5a64f466c8ba5a1761f3bef39 reported cstate change: config changed from index 9 to 11, VOTER a64152fd9595478ca767a9f248ef3b85 (127.9.176.194) evicted. New cstate: current_term: 2 leader_uuid: "543169fd701141c181c55566d2cae00d" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } }
I20240502 01:55:54.684120 27558 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 31de6c641c094cce894d8513b878350c with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:55:54.684358 29615 raft_consensus.cc:2932] T 63136d6964084523bb950cc6568a273b P a64152fd9595478ca767a9f248ef3b85 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } }
I20240502 01:55:54.686246 27981 raft_consensus.cc:1261] T 63136d6964084523bb950cc6568a273b P b550bed5a64f466c8ba5a1761f3bef39 [term 1 FOLLOWER]: Refusing update from remote peer a64152fd9595478ca767a9f248ef3b85: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:54.687350 27827 tablet_service.cc:1514] Processing DeleteTablet for tablet 31de6c641c094cce894d8513b878350c with delete_type TABLET_DATA_TOMBSTONED (TS a64152fd9595478ca767a9f248ef3b85 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:55806
I20240502 01:55:54.687677 29829 consensus_queue.cc:1035] T 63136d6964084523bb950cc6568a273b P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240502 01:55:54.687785 29805 tablet_replica.cc:330] T 31de6c641c094cce894d8513b878350c P a64152fd9595478ca767a9f248ef3b85: stopping tablet replica
I20240502 01:55:54.688532 29805 raft_consensus.cc:2229] T 31de6c641c094cce894d8513b878350c P a64152fd9595478ca767a9f248ef3b85 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:54.688755 29805 raft_consensus.cc:2258] T 31de6c641c094cce894d8513b878350c P a64152fd9595478ca767a9f248ef3b85 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:54.689479 27715 raft_consensus.cc:2932] T 63136d6964084523bb950cc6568a273b P d50f07af87d74f039fb2b43cf166fa43 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } }
I20240502 01:55:54.689875 29805 ts_tablet_manager.cc:1875] T 31de6c641c094cce894d8513b878350c P a64152fd9595478ca767a9f248ef3b85: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:54.695739 27566 catalog_manager.cc:5484] T 63136d6964084523bb950cc6568a273b P d50f07af87d74f039fb2b43cf166fa43 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "a64152fd9595478ca767a9f248ef3b85" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } }
I20240502 01:55:54.698496 29805 ts_tablet_manager.cc:1888] T 31de6c641c094cce894d8513b878350c P a64152fd9595478ca767a9f248ef3b85: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.10
I20240502 01:55:54.698688 29805 log.cc:1196] T 31de6c641c094cce894d8513b878350c P a64152fd9595478ca767a9f248ef3b85: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/wal/wals/31de6c641c094cce894d8513b878350c
I20240502 01:55:54.700985 27560 catalog_manager.cc:4830] TS a64152fd9595478ca767a9f248ef3b85 (127.9.176.194:38689): tablet 31de6c641c094cce894d8513b878350c (table rebalance_test_table_2 [id=d490226c88e64345ad42c7ceb86bf959]) successfully deleted
I20240502 01:55:54.704862 27849 consensus_queue.cc:237] T 63136d6964084523bb950cc6568a273b P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: true } }
I20240502 01:55:54.708020 27715 raft_consensus.cc:1261] T 63136d6964084523bb950cc6568a273b P d50f07af87d74f039fb2b43cf166fa43 [term 1 FOLLOWER]: Refusing update from remote peer a64152fd9595478ca767a9f248ef3b85: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:54.708729 29615 consensus_queue.cc:1035] T 63136d6964084523bb950cc6568a273b P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 01:55:54.709872 27784 consensus_peers.cc:485] T 63136d6964084523bb950cc6568a273b P a64152fd9595478ca767a9f248ef3b85 -> Peer c541378ca18944c49c37f49097bc7f81 (127.9.176.197:32773): Couldn't send request to peer c541378ca18944c49c37f49097bc7f81. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 63136d6964084523bb950cc6568a273b. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:54.710780 27981 raft_consensus.cc:2932] T 63136d6964084523bb950cc6568a273b P b550bed5a64f466c8ba5a1761f3bef39 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } }
I20240502 01:55:54.712862 29832 raft_consensus.cc:2932] T 63136d6964084523bb950cc6568a273b P a64152fd9595478ca767a9f248ef3b85 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c541378ca18944c49c37f49097bc7f81 (127.9.176.197) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: true } } }
I20240502 01:55:54.713550 27715 raft_consensus.cc:2932] T 63136d6964084523bb950cc6568a273b P d50f07af87d74f039fb2b43cf166fa43 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c541378ca18944c49c37f49097bc7f81 (127.9.176.197) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: true } } }
I20240502 01:55:54.715345 27983 raft_consensus.cc:1261] T 63136d6964084523bb950cc6568a273b P b550bed5a64f466c8ba5a1761f3bef39 [term 1 FOLLOWER]: Refusing update from remote peer a64152fd9595478ca767a9f248ef3b85: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:54.717624 29615 consensus_queue.cc:1035] T 63136d6964084523bb950cc6568a273b P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:54.718425 27560 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 63136d6964084523bb950cc6568a273b with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 01:55:54.719947 27572 catalog_manager.cc:5484] T 63136d6964084523bb950cc6568a273b P a64152fd9595478ca767a9f248ef3b85 reported cstate change: config changed from index 4 to 5, NON_VOTER c541378ca18944c49c37f49097bc7f81 (127.9.176.197) added. New cstate: current_term: 1 leader_uuid: "a64152fd9595478ca767a9f248ef3b85" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:55:54.721206 27983 raft_consensus.cc:2932] T 63136d6964084523bb950cc6568a273b P b550bed5a64f466c8ba5a1761f3bef39 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c541378ca18944c49c37f49097bc7f81 (127.9.176.197) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: true } } }
I20240502 01:55:54.936391 28645 consensus_queue.cc:237] T d404c85b11594e7eb1257fcf384beafa P c541378ca18944c49c37f49097bc7f81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } }
I20240502 01:55:54.938339 28778 raft_consensus.cc:1261] T d404c85b11594e7eb1257fcf384beafa P 543169fd701141c181c55566d2cae00d [term 2 FOLLOWER]: Refusing update from remote peer c541378ca18944c49c37f49097bc7f81: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240502 01:55:54.938591 27715 raft_consensus.cc:1261] T d404c85b11594e7eb1257fcf384beafa P d50f07af87d74f039fb2b43cf166fa43 [term 2 FOLLOWER]: Refusing update from remote peer c541378ca18944c49c37f49097bc7f81: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240502 01:55:54.938915 29848 consensus_queue.cc:1035] T d404c85b11594e7eb1257fcf384beafa P c541378ca18944c49c37f49097bc7f81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 01:55:54.939271 29849 consensus_queue.cc:1035] T d404c85b11594e7eb1257fcf384beafa P c541378ca18944c49c37f49097bc7f81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 01:55:54.941810 29755 raft_consensus.cc:2932] T d404c85b11594e7eb1257fcf384beafa P c541378ca18944c49c37f49097bc7f81 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER a64152fd9595478ca767a9f248ef3b85 (127.9.176.194) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } }
I20240502 01:55:54.942631 27715 raft_consensus.cc:2932] T d404c85b11594e7eb1257fcf384beafa P d50f07af87d74f039fb2b43cf166fa43 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER a64152fd9595478ca767a9f248ef3b85 (127.9.176.194) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } }
I20240502 01:55:54.945112 27560 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d404c85b11594e7eb1257fcf384beafa with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:55:54.945789 28778 raft_consensus.cc:2932] T d404c85b11594e7eb1257fcf384beafa P 543169fd701141c181c55566d2cae00d [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER a64152fd9595478ca767a9f248ef3b85 (127.9.176.194) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } }
I20240502 01:55:54.947170 27566 catalog_manager.cc:5484] T d404c85b11594e7eb1257fcf384beafa P c541378ca18944c49c37f49097bc7f81 reported cstate change: config changed from index 9 to 11, VOTER a64152fd9595478ca767a9f248ef3b85 (127.9.176.194) evicted. New cstate: current_term: 2 leader_uuid: "c541378ca18944c49c37f49097bc7f81" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:55:54.951617 27827 tablet_service.cc:1514] Processing DeleteTablet for tablet d404c85b11594e7eb1257fcf384beafa with delete_type TABLET_DATA_TOMBSTONED (TS a64152fd9595478ca767a9f248ef3b85 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:55806
I20240502 01:55:54.952068 29805 tablet_replica.cc:330] T d404c85b11594e7eb1257fcf384beafa P a64152fd9595478ca767a9f248ef3b85: stopping tablet replica
I20240502 01:55:54.952292 29805 raft_consensus.cc:2229] T d404c85b11594e7eb1257fcf384beafa P a64152fd9595478ca767a9f248ef3b85 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:54.952549 29805 raft_consensus.cc:2258] T d404c85b11594e7eb1257fcf384beafa P a64152fd9595478ca767a9f248ef3b85 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:54.953445 29805 ts_tablet_manager.cc:1875] T d404c85b11594e7eb1257fcf384beafa P a64152fd9595478ca767a9f248ef3b85: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:54.959543 29805 ts_tablet_manager.cc:1888] T d404c85b11594e7eb1257fcf384beafa P a64152fd9595478ca767a9f248ef3b85: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.10
I20240502 01:55:54.959784 29805 log.cc:1196] T d404c85b11594e7eb1257fcf384beafa P a64152fd9595478ca767a9f248ef3b85: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/wal/wals/d404c85b11594e7eb1257fcf384beafa
I20240502 01:55:54.960662 27560 catalog_manager.cc:4830] TS a64152fd9595478ca767a9f248ef3b85 (127.9.176.194:38689): tablet d404c85b11594e7eb1257fcf384beafa (table rebalance_test_table_2 [id=d490226c88e64345ad42c7ceb86bf959]) successfully deleted
I20240502 01:55:54.965772 27849 consensus_queue.cc:237] T 8a29996e3d594ebebae8b5fe7e02aa8c P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } }
I20240502 01:55:54.967993 27715 raft_consensus.cc:1261] T 8a29996e3d594ebebae8b5fe7e02aa8c P d50f07af87d74f039fb2b43cf166fa43 [term 1 FOLLOWER]: Refusing update from remote peer a64152fd9595478ca767a9f248ef3b85: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:54.968026 27983 raft_consensus.cc:1261] T 8a29996e3d594ebebae8b5fe7e02aa8c P b550bed5a64f466c8ba5a1761f3bef39 [term 1 FOLLOWER]: Refusing update from remote peer a64152fd9595478ca767a9f248ef3b85: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:54.968616 29615 consensus_queue.cc:1035] T 8a29996e3d594ebebae8b5fe7e02aa8c P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:54.968940 29829 consensus_queue.cc:1035] T 8a29996e3d594ebebae8b5fe7e02aa8c P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:54.971750 29615 raft_consensus.cc:2932] T 8a29996e3d594ebebae8b5fe7e02aa8c P a64152fd9595478ca767a9f248ef3b85 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } }
I20240502 01:55:54.972608 27715 raft_consensus.cc:2932] T 8a29996e3d594ebebae8b5fe7e02aa8c P d50f07af87d74f039fb2b43cf166fa43 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } }
I20240502 01:55:54.975809 27983 raft_consensus.cc:2932] T 8a29996e3d594ebebae8b5fe7e02aa8c P b550bed5a64f466c8ba5a1761f3bef39 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } }
I20240502 01:55:54.978658 27572 catalog_manager.cc:5484] T 8a29996e3d594ebebae8b5fe7e02aa8c P a64152fd9595478ca767a9f248ef3b85 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "a64152fd9595478ca767a9f248ef3b85" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } health_report { overall_health: HEALTHY } } }
I20240502 01:55:54.985455 27849 consensus_queue.cc:237] T 8a29996e3d594ebebae8b5fe7e02aa8c P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: true } }
I20240502 01:55:54.989219 27715 raft_consensus.cc:1261] T 8a29996e3d594ebebae8b5fe7e02aa8c P d50f07af87d74f039fb2b43cf166fa43 [term 1 FOLLOWER]: Refusing update from remote peer a64152fd9595478ca767a9f248ef3b85: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:54.989679 27983 raft_consensus.cc:1261] T 8a29996e3d594ebebae8b5fe7e02aa8c P b550bed5a64f466c8ba5a1761f3bef39 [term 1 FOLLOWER]: Refusing update from remote peer a64152fd9595478ca767a9f248ef3b85: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240502 01:55:54.990154 27784 consensus_peers.cc:485] T 8a29996e3d594ebebae8b5fe7e02aa8c P a64152fd9595478ca767a9f248ef3b85 -> Peer c541378ca18944c49c37f49097bc7f81 (127.9.176.197:32773): Couldn't send request to peer c541378ca18944c49c37f49097bc7f81. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8a29996e3d594ebebae8b5fe7e02aa8c. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:54.990532 29615 consensus_queue.cc:1035] T 8a29996e3d594ebebae8b5fe7e02aa8c P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:54.991072 29829 consensus_queue.cc:1035] T 8a29996e3d594ebebae8b5fe7e02aa8c P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:54.993685 29615 raft_consensus.cc:2932] T 8a29996e3d594ebebae8b5fe7e02aa8c P a64152fd9595478ca767a9f248ef3b85 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c541378ca18944c49c37f49097bc7f81 (127.9.176.197) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: true } } }
I20240502 01:55:54.994735 27983 raft_consensus.cc:2932] T 8a29996e3d594ebebae8b5fe7e02aa8c P b550bed5a64f466c8ba5a1761f3bef39 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c541378ca18944c49c37f49097bc7f81 (127.9.176.197) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: true } } }
I20240502 01:55:54.994942 27715 raft_consensus.cc:2932] T 8a29996e3d594ebebae8b5fe7e02aa8c P d50f07af87d74f039fb2b43cf166fa43 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c541378ca18944c49c37f49097bc7f81 (127.9.176.197) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: true } } }
I20240502 01:55:54.998670 27560 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8a29996e3d594ebebae8b5fe7e02aa8c with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 01:55:55.002187 27569 catalog_manager.cc:5484] T 8a29996e3d594ebebae8b5fe7e02aa8c P b550bed5a64f466c8ba5a1761f3bef39 reported cstate change: config changed from index 4 to 5, NON_VOTER c541378ca18944c49c37f49097bc7f81 (127.9.176.197) added. New cstate: current_term: 1 leader_uuid: "a64152fd9595478ca767a9f248ef3b85" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: true } } }
I20240502 01:55:55.221952 29873 ts_tablet_manager.cc:902] T 63136d6964084523bb950cc6568a273b P c541378ca18944c49c37f49097bc7f81: Initiating tablet copy from peer a64152fd9595478ca767a9f248ef3b85 (127.9.176.194:38689)
I20240502 01:55:55.223255 29873 tablet_copy_client.cc:308] T 63136d6964084523bb950cc6568a273b P c541378ca18944c49c37f49097bc7f81: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.194:38689
I20240502 01:55:55.223948 27869 tablet_copy_service.cc:140] P a64152fd9595478ca767a9f248ef3b85: Received BeginTabletCopySession request for tablet 63136d6964084523bb950cc6568a273b from peer c541378ca18944c49c37f49097bc7f81 ({username='slave'} at 127.9.176.197:47985)
I20240502 01:55:55.224131 27869 tablet_copy_service.cc:161] P a64152fd9595478ca767a9f248ef3b85: Beginning new tablet copy session on tablet 63136d6964084523bb950cc6568a273b from peer c541378ca18944c49c37f49097bc7f81 at {username='slave'} at 127.9.176.197:47985: session id = c541378ca18944c49c37f49097bc7f81-63136d6964084523bb950cc6568a273b
I20240502 01:55:55.226050 27869 tablet_copy_source_session.cc:205] T 63136d6964084523bb950cc6568a273b P a64152fd9595478ca767a9f248ef3b85: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:55.227234 29873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63136d6964084523bb950cc6568a273b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:55.231551 29873 tablet_copy_client.cc:783] T 63136d6964084523bb950cc6568a273b P c541378ca18944c49c37f49097bc7f81: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:55.231850 29873 tablet_copy_client.cc:647] T 63136d6964084523bb950cc6568a273b P c541378ca18944c49c37f49097bc7f81: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:55.233559 29873 tablet_copy_client.cc:523] T 63136d6964084523bb950cc6568a273b P c541378ca18944c49c37f49097bc7f81: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:55.236518 29873 tablet_bootstrap.cc:492] T 63136d6964084523bb950cc6568a273b P c541378ca18944c49c37f49097bc7f81: Bootstrap starting.
I20240502 01:55:55.243806 27849 consensus_queue.cc:237] T 5e8b27dca9af473188c50269820343f1 P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } }
I20240502 01:55:55.246136 28645 raft_consensus.cc:1261] T 5e8b27dca9af473188c50269820343f1 P c541378ca18944c49c37f49097bc7f81 [term 1 FOLLOWER]: Refusing update from remote peer a64152fd9595478ca767a9f248ef3b85: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:55.246253 27983 raft_consensus.cc:1261] T 5e8b27dca9af473188c50269820343f1 P b550bed5a64f466c8ba5a1761f3bef39 [term 1 FOLLOWER]: Refusing update from remote peer a64152fd9595478ca767a9f248ef3b85: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:55.246747 29615 consensus_queue.cc:1035] T 5e8b27dca9af473188c50269820343f1 P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:55.247069 29829 consensus_queue.cc:1035] T 5e8b27dca9af473188c50269820343f1 P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:55.250089 29832 raft_consensus.cc:2932] T 5e8b27dca9af473188c50269820343f1 P a64152fd9595478ca767a9f248ef3b85 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } }
I20240502 01:55:55.251904 27983 raft_consensus.cc:2932] T 5e8b27dca9af473188c50269820343f1 P b550bed5a64f466c8ba5a1761f3bef39 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } }
I20240502 01:55:55.254854 27566 catalog_manager.cc:5484] T 5e8b27dca9af473188c50269820343f1 P a64152fd9595478ca767a9f248ef3b85 reported cstate change: config changed from index 6 to 7. New cstate: current_term: 1 leader_uuid: "a64152fd9595478ca767a9f248ef3b85" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:55:55.258790 28645 raft_consensus.cc:2932] T 5e8b27dca9af473188c50269820343f1 P c541378ca18944c49c37f49097bc7f81 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } }
I20240502 01:55:55.263963 27849 consensus_queue.cc:237] T 5e8b27dca9af473188c50269820343f1 P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: true } }
I20240502 01:55:55.266674 28646 raft_consensus.cc:1261] T 5e8b27dca9af473188c50269820343f1 P c541378ca18944c49c37f49097bc7f81 [term 1 FOLLOWER]: Refusing update from remote peer a64152fd9595478ca767a9f248ef3b85: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240502 01:55:55.267194 27782 consensus_peers.cc:485] T 5e8b27dca9af473188c50269820343f1 P a64152fd9595478ca767a9f248ef3b85 -> Peer 543169fd701141c181c55566d2cae00d (127.9.176.196:38773): Couldn't send request to peer 543169fd701141c181c55566d2cae00d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5e8b27dca9af473188c50269820343f1. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:55.267386 29832 consensus_queue.cc:1035] T 5e8b27dca9af473188c50269820343f1 P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:55.267802 27983 raft_consensus.cc:1261] T 5e8b27dca9af473188c50269820343f1 P b550bed5a64f466c8ba5a1761f3bef39 [term 1 FOLLOWER]: Refusing update from remote peer a64152fd9595478ca767a9f248ef3b85: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:55.270313 29832 consensus_queue.cc:1035] T 5e8b27dca9af473188c50269820343f1 P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:55.271099 29829 raft_consensus.cc:2932] T 5e8b27dca9af473188c50269820343f1 P a64152fd9595478ca767a9f248ef3b85 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 543169fd701141c181c55566d2cae00d (127.9.176.196) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: true } } }
I20240502 01:55:55.274220 27560 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5e8b27dca9af473188c50269820343f1 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 01:55:55.275040 28646 raft_consensus.cc:2932] T 5e8b27dca9af473188c50269820343f1 P c541378ca18944c49c37f49097bc7f81 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 543169fd701141c181c55566d2cae00d (127.9.176.196) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: true } } }
I20240502 01:55:55.281265 27981 raft_consensus.cc:2932] T 5e8b27dca9af473188c50269820343f1 P b550bed5a64f466c8ba5a1761f3bef39 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 543169fd701141c181c55566d2cae00d (127.9.176.196) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: true } } }
I20240502 01:55:55.284806 27566 catalog_manager.cc:5484] T 5e8b27dca9af473188c50269820343f1 P a64152fd9595478ca767a9f248ef3b85 reported cstate change: config changed from index 7 to 8, NON_VOTER 543169fd701141c181c55566d2cae00d (127.9.176.196) added. New cstate: current_term: 1 leader_uuid: "a64152fd9595478ca767a9f248ef3b85" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:55:55.298856 29873 tablet_bootstrap.cc:492] T 63136d6964084523bb950cc6568a273b P c541378ca18944c49c37f49097bc7f81: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:55.299499 29873 tablet_bootstrap.cc:492] T 63136d6964084523bb950cc6568a273b P c541378ca18944c49c37f49097bc7f81: Bootstrap complete.
I20240502 01:55:55.299926 29873 ts_tablet_manager.cc:1367] T 63136d6964084523bb950cc6568a273b P c541378ca18944c49c37f49097bc7f81: Time spent bootstrapping tablet: real 0.063s	user 0.052s	sys 0.005s
I20240502 01:55:55.300985 29873 raft_consensus.cc:348] T 63136d6964084523bb950cc6568a273b P c541378ca18944c49c37f49097bc7f81 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: true } }
I20240502 01:55:55.301326 29873 raft_consensus.cc:729] T 63136d6964084523bb950cc6568a273b P c541378ca18944c49c37f49097bc7f81 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c541378ca18944c49c37f49097bc7f81, State: Initialized, Role: LEARNER
I20240502 01:55:55.301561 29873 consensus_queue.cc:260] T 63136d6964084523bb950cc6568a273b P c541378ca18944c49c37f49097bc7f81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: true } }
I20240502 01:55:55.302186 29873 ts_tablet_manager.cc:1398] T 63136d6964084523bb950cc6568a273b P c541378ca18944c49c37f49097bc7f81: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 01:55:55.303071 27869 tablet_copy_service.cc:342] P a64152fd9595478ca767a9f248ef3b85: Request end of tablet copy session c541378ca18944c49c37f49097bc7f81-63136d6964084523bb950cc6568a273b received from {username='slave'} at 127.9.176.197:47985
I20240502 01:55:55.303193 27869 tablet_copy_service.cc:433] P a64152fd9595478ca767a9f248ef3b85: ending tablet copy session c541378ca18944c49c37f49097bc7f81-63136d6964084523bb950cc6568a273b on tablet 63136d6964084523bb950cc6568a273b with peer c541378ca18944c49c37f49097bc7f81
I20240502 01:55:55.374847 29873 ts_tablet_manager.cc:902] T 8a29996e3d594ebebae8b5fe7e02aa8c P c541378ca18944c49c37f49097bc7f81: Initiating tablet copy from peer a64152fd9595478ca767a9f248ef3b85 (127.9.176.194:38689)
I20240502 01:55:55.376255 29873 tablet_copy_client.cc:308] T 8a29996e3d594ebebae8b5fe7e02aa8c P c541378ca18944c49c37f49097bc7f81: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.194:38689
I20240502 01:55:55.377089 27869 tablet_copy_service.cc:140] P a64152fd9595478ca767a9f248ef3b85: Received BeginTabletCopySession request for tablet 8a29996e3d594ebebae8b5fe7e02aa8c from peer c541378ca18944c49c37f49097bc7f81 ({username='slave'} at 127.9.176.197:47985)
I20240502 01:55:55.377261 27869 tablet_copy_service.cc:161] P a64152fd9595478ca767a9f248ef3b85: Beginning new tablet copy session on tablet 8a29996e3d594ebebae8b5fe7e02aa8c from peer c541378ca18944c49c37f49097bc7f81 at {username='slave'} at 127.9.176.197:47985: session id = c541378ca18944c49c37f49097bc7f81-8a29996e3d594ebebae8b5fe7e02aa8c
I20240502 01:55:55.379480 27869 tablet_copy_source_session.cc:205] T 8a29996e3d594ebebae8b5fe7e02aa8c P a64152fd9595478ca767a9f248ef3b85: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:55.380842 29873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a29996e3d594ebebae8b5fe7e02aa8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:55.385895 29873 tablet_copy_client.cc:783] T 8a29996e3d594ebebae8b5fe7e02aa8c P c541378ca18944c49c37f49097bc7f81: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:55.386157 29873 tablet_copy_client.cc:647] T 8a29996e3d594ebebae8b5fe7e02aa8c P c541378ca18944c49c37f49097bc7f81: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:55.387989 29873 tablet_copy_client.cc:523] T 8a29996e3d594ebebae8b5fe7e02aa8c P c541378ca18944c49c37f49097bc7f81: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:55.390507 29873 tablet_bootstrap.cc:492] T 8a29996e3d594ebebae8b5fe7e02aa8c P c541378ca18944c49c37f49097bc7f81: Bootstrap starting.
I20240502 01:55:55.421859 29873 tablet_bootstrap.cc:492] T 8a29996e3d594ebebae8b5fe7e02aa8c P c541378ca18944c49c37f49097bc7f81: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:55.422315 29873 tablet_bootstrap.cc:492] T 8a29996e3d594ebebae8b5fe7e02aa8c P c541378ca18944c49c37f49097bc7f81: Bootstrap complete.
I20240502 01:55:55.422566 29873 ts_tablet_manager.cc:1367] T 8a29996e3d594ebebae8b5fe7e02aa8c P c541378ca18944c49c37f49097bc7f81: Time spent bootstrapping tablet: real 0.032s	user 0.030s	sys 0.001s
I20240502 01:55:55.423270 29873 raft_consensus.cc:348] T 8a29996e3d594ebebae8b5fe7e02aa8c P c541378ca18944c49c37f49097bc7f81 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: true } }
I20240502 01:55:55.423481 29873 raft_consensus.cc:729] T 8a29996e3d594ebebae8b5fe7e02aa8c P c541378ca18944c49c37f49097bc7f81 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c541378ca18944c49c37f49097bc7f81, State: Initialized, Role: LEARNER
I20240502 01:55:55.423707 29873 consensus_queue.cc:260] T 8a29996e3d594ebebae8b5fe7e02aa8c P c541378ca18944c49c37f49097bc7f81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: true } }
I20240502 01:55:55.424216 29873 ts_tablet_manager.cc:1398] T 8a29996e3d594ebebae8b5fe7e02aa8c P c541378ca18944c49c37f49097bc7f81: Time spent starting tablet: real 0.002s	user 0.004s	sys 0.000s
I20240502 01:55:55.425084 27869 tablet_copy_service.cc:342] P a64152fd9595478ca767a9f248ef3b85: Request end of tablet copy session c541378ca18944c49c37f49097bc7f81-8a29996e3d594ebebae8b5fe7e02aa8c received from {username='slave'} at 127.9.176.197:47985
I20240502 01:55:55.425268 27869 tablet_copy_service.cc:433] P a64152fd9595478ca767a9f248ef3b85: ending tablet copy session c541378ca18944c49c37f49097bc7f81-8a29996e3d594ebebae8b5fe7e02aa8c on tablet 8a29996e3d594ebebae8b5fe7e02aa8c with peer c541378ca18944c49c37f49097bc7f81
I20240502 01:55:55.649652 28646 raft_consensus.cc:1203] T 63136d6964084523bb950cc6568a273b P c541378ca18944c49c37f49097bc7f81 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
I20240502 01:55:55.745414 29923 ts_tablet_manager.cc:902] T 5e8b27dca9af473188c50269820343f1 P 543169fd701141c181c55566d2cae00d: Initiating tablet copy from peer a64152fd9595478ca767a9f248ef3b85 (127.9.176.194:38689)
I20240502 01:55:55.746526 29923 tablet_copy_client.cc:308] T 5e8b27dca9af473188c50269820343f1 P 543169fd701141c181c55566d2cae00d: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.194:38689
I20240502 01:55:55.747242 27869 tablet_copy_service.cc:140] P a64152fd9595478ca767a9f248ef3b85: Received BeginTabletCopySession request for tablet 5e8b27dca9af473188c50269820343f1 from peer 543169fd701141c181c55566d2cae00d ({username='slave'} at 127.9.176.196:38793)
I20240502 01:55:55.747433 27869 tablet_copy_service.cc:161] P a64152fd9595478ca767a9f248ef3b85: Beginning new tablet copy session on tablet 5e8b27dca9af473188c50269820343f1 from peer 543169fd701141c181c55566d2cae00d at {username='slave'} at 127.9.176.196:38793: session id = 543169fd701141c181c55566d2cae00d-5e8b27dca9af473188c50269820343f1
I20240502 01:55:55.749521 27869 tablet_copy_source_session.cc:205] T 5e8b27dca9af473188c50269820343f1 P a64152fd9595478ca767a9f248ef3b85: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:55.750774 29923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e8b27dca9af473188c50269820343f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:55.755530 29923 tablet_copy_client.cc:783] T 5e8b27dca9af473188c50269820343f1 P 543169fd701141c181c55566d2cae00d: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:55.755824 29923 tablet_copy_client.cc:647] T 5e8b27dca9af473188c50269820343f1 P 543169fd701141c181c55566d2cae00d: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:55.757452 29923 tablet_copy_client.cc:523] T 5e8b27dca9af473188c50269820343f1 P 543169fd701141c181c55566d2cae00d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:55.760491 29923 tablet_bootstrap.cc:492] T 5e8b27dca9af473188c50269820343f1 P 543169fd701141c181c55566d2cae00d: Bootstrap starting.
I20240502 01:55:55.801304 29923 tablet_bootstrap.cc:492] T 5e8b27dca9af473188c50269820343f1 P 543169fd701141c181c55566d2cae00d: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:55.801770 29923 tablet_bootstrap.cc:492] T 5e8b27dca9af473188c50269820343f1 P 543169fd701141c181c55566d2cae00d: Bootstrap complete.
I20240502 01:55:55.802057 29923 ts_tablet_manager.cc:1367] T 5e8b27dca9af473188c50269820343f1 P 543169fd701141c181c55566d2cae00d: Time spent bootstrapping tablet: real 0.042s	user 0.028s	sys 0.011s
I20240502 01:55:55.802755 29923 raft_consensus.cc:348] T 5e8b27dca9af473188c50269820343f1 P 543169fd701141c181c55566d2cae00d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: true } }
I20240502 01:55:55.802984 29923 raft_consensus.cc:729] T 5e8b27dca9af473188c50269820343f1 P 543169fd701141c181c55566d2cae00d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 543169fd701141c181c55566d2cae00d, State: Initialized, Role: LEARNER
I20240502 01:55:55.803182 29923 consensus_queue.cc:260] T 5e8b27dca9af473188c50269820343f1 P 543169fd701141c181c55566d2cae00d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: true } }
I20240502 01:55:55.803748 29923 ts_tablet_manager.cc:1398] T 5e8b27dca9af473188c50269820343f1 P 543169fd701141c181c55566d2cae00d: Time spent starting tablet: real 0.002s	user 0.004s	sys 0.000s
I20240502 01:55:55.804615 27869 tablet_copy_service.cc:342] P a64152fd9595478ca767a9f248ef3b85: Request end of tablet copy session 543169fd701141c181c55566d2cae00d-5e8b27dca9af473188c50269820343f1 received from {username='slave'} at 127.9.176.196:38793
I20240502 01:55:55.804744 27869 tablet_copy_service.cc:433] P a64152fd9595478ca767a9f248ef3b85: ending tablet copy session 543169fd701141c181c55566d2cae00d-5e8b27dca9af473188c50269820343f1 on tablet 5e8b27dca9af473188c50269820343f1 with peer 543169fd701141c181c55566d2cae00d
I20240502 01:55:55.929262 28646 raft_consensus.cc:1203] T 8a29996e3d594ebebae8b5fe7e02aa8c P c541378ca18944c49c37f49097bc7f81 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
I20240502 01:55:56.183359 28778 raft_consensus.cc:1203] T 5e8b27dca9af473188c50269820343f1 P 543169fd701141c181c55566d2cae00d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8]   Dedup: 1.8->[]
I20240502 01:55:56.185052 29829 raft_consensus.cc:1050] T 63136d6964084523bb950cc6568a273b P a64152fd9595478ca767a9f248ef3b85: attempting to promote NON_VOTER c541378ca18944c49c37f49097bc7f81 to VOTER
I20240502 01:55:56.186103 29829 consensus_queue.cc:237] T 63136d6964084523bb950cc6568a273b P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } }
I20240502 01:55:56.189019 28646 raft_consensus.cc:1261] T 63136d6964084523bb950cc6568a273b P c541378ca18944c49c37f49097bc7f81 [term 1 LEARNER]: Refusing update from remote peer a64152fd9595478ca767a9f248ef3b85: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:56.189057 27715 raft_consensus.cc:1261] T 63136d6964084523bb950cc6568a273b P d50f07af87d74f039fb2b43cf166fa43 [term 1 FOLLOWER]: Refusing update from remote peer a64152fd9595478ca767a9f248ef3b85: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:56.189087 27983 raft_consensus.cc:1261] T 63136d6964084523bb950cc6568a273b P b550bed5a64f466c8ba5a1761f3bef39 [term 1 FOLLOWER]: Refusing update from remote peer a64152fd9595478ca767a9f248ef3b85: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:56.189702 29829 consensus_queue.cc:1035] T 63136d6964084523bb950cc6568a273b P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:56.190085 29961 consensus_queue.cc:1035] T 63136d6964084523bb950cc6568a273b P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:56.190461 29832 consensus_queue.cc:1035] T 63136d6964084523bb950cc6568a273b P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:56.199465 29829 raft_consensus.cc:2932] T 63136d6964084523bb950cc6568a273b P a64152fd9595478ca767a9f248ef3b85 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c541378ca18944c49c37f49097bc7f81 (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } }
I20240502 01:55:56.200481 27715 raft_consensus.cc:2932] T 63136d6964084523bb950cc6568a273b P d50f07af87d74f039fb2b43cf166fa43 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c541378ca18944c49c37f49097bc7f81 (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } }
I20240502 01:55:56.200357 28646 raft_consensus.cc:2932] T 63136d6964084523bb950cc6568a273b P c541378ca18944c49c37f49097bc7f81 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c541378ca18944c49c37f49097bc7f81 (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } }
I20240502 01:55:56.205778 27566 catalog_manager.cc:5484] T 63136d6964084523bb950cc6568a273b P d50f07af87d74f039fb2b43cf166fa43 reported cstate change: config changed from index 5 to 6, c541378ca18944c49c37f49097bc7f81 (127.9.176.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a64152fd9595478ca767a9f248ef3b85" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } }
I20240502 01:55:56.206599 27983 raft_consensus.cc:2932] T 63136d6964084523bb950cc6568a273b P b550bed5a64f466c8ba5a1761f3bef39 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c541378ca18944c49c37f49097bc7f81 (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } }
I20240502 01:55:56.260192 27849 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "63136d6964084523bb950cc6568a273b"
dest_uuid: "a64152fd9595478ca767a9f248ef3b85"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:53346
I20240502 01:55:56.260440 27849 raft_consensus.cc:595] T 63136d6964084523bb950cc6568a273b P a64152fd9595478ca767a9f248ef3b85 [term 1 LEADER]: Received request to transfer leadership
I20240502 01:55:56.328620 29961 raft_consensus.cc:1050] T 8a29996e3d594ebebae8b5fe7e02aa8c P a64152fd9595478ca767a9f248ef3b85: attempting to promote NON_VOTER c541378ca18944c49c37f49097bc7f81 to VOTER
I20240502 01:55:56.329388 29961 consensus_queue.cc:237] T 8a29996e3d594ebebae8b5fe7e02aa8c P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } }
I20240502 01:55:56.331786 28646 raft_consensus.cc:1261] T 8a29996e3d594ebebae8b5fe7e02aa8c P c541378ca18944c49c37f49097bc7f81 [term 1 LEARNER]: Refusing update from remote peer a64152fd9595478ca767a9f248ef3b85: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:56.331898 27983 raft_consensus.cc:1261] T 8a29996e3d594ebebae8b5fe7e02aa8c P b550bed5a64f466c8ba5a1761f3bef39 [term 1 FOLLOWER]: Refusing update from remote peer a64152fd9595478ca767a9f248ef3b85: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:56.331919 27715 raft_consensus.cc:1261] T 8a29996e3d594ebebae8b5fe7e02aa8c P d50f07af87d74f039fb2b43cf166fa43 [term 1 FOLLOWER]: Refusing update from remote peer a64152fd9595478ca767a9f248ef3b85: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:56.332425 29961 consensus_queue.cc:1035] T 8a29996e3d594ebebae8b5fe7e02aa8c P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:56.332758 29829 consensus_queue.cc:1035] T 8a29996e3d594ebebae8b5fe7e02aa8c P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:56.333016 29615 consensus_queue.cc:1035] T 8a29996e3d594ebebae8b5fe7e02aa8c P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:56.336750 29829 raft_consensus.cc:2932] T 8a29996e3d594ebebae8b5fe7e02aa8c P a64152fd9595478ca767a9f248ef3b85 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c541378ca18944c49c37f49097bc7f81 (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } }
I20240502 01:55:56.337575 28646 raft_consensus.cc:2932] T 8a29996e3d594ebebae8b5fe7e02aa8c P c541378ca18944c49c37f49097bc7f81 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c541378ca18944c49c37f49097bc7f81 (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } }
I20240502 01:55:56.342224 27715 raft_consensus.cc:2932] T 8a29996e3d594ebebae8b5fe7e02aa8c P d50f07af87d74f039fb2b43cf166fa43 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c541378ca18944c49c37f49097bc7f81 (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } }
I20240502 01:55:56.342816 27569 catalog_manager.cc:5484] T 8a29996e3d594ebebae8b5fe7e02aa8c P a64152fd9595478ca767a9f248ef3b85 reported cstate change: config changed from index 5 to 6, c541378ca18944c49c37f49097bc7f81 (127.9.176.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a64152fd9595478ca767a9f248ef3b85" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:55:56.345504 27983 raft_consensus.cc:2932] T 8a29996e3d594ebebae8b5fe7e02aa8c P b550bed5a64f466c8ba5a1761f3bef39 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c541378ca18944c49c37f49097bc7f81 (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } }
I20240502 01:55:56.493745 27849 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8a29996e3d594ebebae8b5fe7e02aa8c"
dest_uuid: "a64152fd9595478ca767a9f248ef3b85"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:53382
I20240502 01:55:56.493973 27849 raft_consensus.cc:595] T 8a29996e3d594ebebae8b5fe7e02aa8c P a64152fd9595478ca767a9f248ef3b85 [term 1 LEADER]: Received request to transfer leadership
I20240502 01:55:56.512612 27849 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "63136d6964084523bb950cc6568a273b"
dest_uuid: "a64152fd9595478ca767a9f248ef3b85"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:53396
I20240502 01:55:56.512853 27849 raft_consensus.cc:595] T 63136d6964084523bb950cc6568a273b P a64152fd9595478ca767a9f248ef3b85 [term 1 LEADER]: Received request to transfer leadership
I20240502 01:55:56.600857 29961 raft_consensus.cc:979] T 63136d6964084523bb950cc6568a273b P a64152fd9595478ca767a9f248ef3b85: : Instructing follower c541378ca18944c49c37f49097bc7f81 to start an election
I20240502 01:55:56.601013 29961 raft_consensus.cc:1067] T 63136d6964084523bb950cc6568a273b P a64152fd9595478ca767a9f248ef3b85 [term 1 LEADER]: Signalling peer c541378ca18944c49c37f49097bc7f81 to start an election
I20240502 01:55:56.601722 28646 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "63136d6964084523bb950cc6568a273b"
dest_uuid: "c541378ca18944c49c37f49097bc7f81"
 from {username='slave'} at 127.9.176.194:59883
I20240502 01:55:56.601927 28646 raft_consensus.cc:482] T 63136d6964084523bb950cc6568a273b P c541378ca18944c49c37f49097bc7f81 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:55:56.602057 28646 raft_consensus.cc:3037] T 63136d6964084523bb950cc6568a273b P c541378ca18944c49c37f49097bc7f81 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:55:56.604049 28646 raft_consensus.cc:504] T 63136d6964084523bb950cc6568a273b P c541378ca18944c49c37f49097bc7f81 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } }
I20240502 01:55:56.604758 28646 leader_election.cc:288] T 63136d6964084523bb950cc6568a273b P c541378ca18944c49c37f49097bc7f81 [CANDIDATE]: Term 2 election: Requested vote from peers d50f07af87d74f039fb2b43cf166fa43 (127.9.176.193:34203), a64152fd9595478ca767a9f248ef3b85 (127.9.176.194:38689), b550bed5a64f466c8ba5a1761f3bef39 (127.9.176.195:40013)
I20240502 01:55:56.605247 27849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63136d6964084523bb950cc6568a273b" candidate_uuid: "c541378ca18944c49c37f49097bc7f81" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "a64152fd9595478ca767a9f248ef3b85"
I20240502 01:55:56.605365 27983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63136d6964084523bb950cc6568a273b" candidate_uuid: "c541378ca18944c49c37f49097bc7f81" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "b550bed5a64f466c8ba5a1761f3bef39"
I20240502 01:55:56.605530 27849 raft_consensus.cc:3032] T 63136d6964084523bb950cc6568a273b P a64152fd9595478ca767a9f248ef3b85 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:55:56.605489 27715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63136d6964084523bb950cc6568a273b" candidate_uuid: "c541378ca18944c49c37f49097bc7f81" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "d50f07af87d74f039fb2b43cf166fa43"
I20240502 01:55:56.605587 27983 raft_consensus.cc:3037] T 63136d6964084523bb950cc6568a273b P b550bed5a64f466c8ba5a1761f3bef39 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:55:56.605608 27849 raft_consensus.cc:729] T 63136d6964084523bb950cc6568a273b P a64152fd9595478ca767a9f248ef3b85 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a64152fd9595478ca767a9f248ef3b85, State: Running, Role: LEADER
I20240502 01:55:56.605707 27715 raft_consensus.cc:3037] T 63136d6964084523bb950cc6568a273b P d50f07af87d74f039fb2b43cf166fa43 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:55:56.605937 27849 consensus_queue.cc:260] T 63136d6964084523bb950cc6568a273b P a64152fd9595478ca767a9f248ef3b85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } }
I20240502 01:55:56.606390 27849 raft_consensus.cc:3037] T 63136d6964084523bb950cc6568a273b P a64152fd9595478ca767a9f248ef3b85 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:55:56.607656 27983 raft_consensus.cc:2454] T 63136d6964084523bb950cc6568a273b P b550bed5a64f466c8ba5a1761f3bef39 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c541378ca18944c49c37f49097bc7f81 in term 2.
I20240502 01:55:56.608544 27715 raft_consensus.cc:2454] T 63136d6964084523bb950cc6568a273b P d50f07af87d74f039fb2b43cf166fa43 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c541378ca18944c49c37f49097bc7f81 in term 2.
I20240502 01:55:56.609023 28579 leader_election.cc:302] T 63136d6964084523bb950cc6568a273b P c541378ca18944c49c37f49097bc7f81 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: b550bed5a64f466c8ba5a1761f3bef39, c541378ca18944c49c37f49097bc7f81, d50f07af87d74f039fb2b43cf166fa43; no voters: 
I20240502 01:55:56.609050 27849 raft_consensus.cc:2454] T 63136d6964084523bb950cc6568a273b P a64152fd9595478ca767a9f248ef3b85 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c541378ca18944c49c37f49097bc7f81 in term 2.
I20240502 01:55:56.609324 29755 raft_consensus.cc:2781] T 63136d6964084523bb950cc6568a273b P c541378ca18944c49c37f49097bc7f81 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:55:56.609709 29755 raft_consensus.cc:686] T 63136d6964084523bb950cc6568a273b P c541378ca18944c49c37f49097bc7f81 [term 2 LEADER]: Becoming Leader. State: Replica: c541378ca18944c49c37f49097bc7f81, State: Running, Role: LEADER
I20240502 01:55:56.610051 29755 consensus_queue.cc:237] T 63136d6964084523bb950cc6568a273b P c541378ca18944c49c37f49097bc7f81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } }
I20240502 01:55:56.612746 27569 catalog_manager.cc:5484] T 63136d6964084523bb950cc6568a273b P c541378ca18944c49c37f49097bc7f81 reported cstate change: term changed from 1 to 2, leader changed from a64152fd9595478ca767a9f248ef3b85 (127.9.176.194) to c541378ca18944c49c37f49097bc7f81 (127.9.176.197). New cstate: current_term: 2 leader_uuid: "c541378ca18944c49c37f49097bc7f81" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:55:56.633653 29961 raft_consensus.cc:1050] T 5e8b27dca9af473188c50269820343f1 P a64152fd9595478ca767a9f248ef3b85: attempting to promote NON_VOTER 543169fd701141c181c55566d2cae00d to VOTER
I20240502 01:55:56.634364 29961 consensus_queue.cc:237] T 5e8b27dca9af473188c50269820343f1 P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } }
I20240502 01:55:56.636641 28778 raft_consensus.cc:1261] T 5e8b27dca9af473188c50269820343f1 P 543169fd701141c181c55566d2cae00d [term 1 LEARNER]: Refusing update from remote peer a64152fd9595478ca767a9f248ef3b85: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:56.636664 27983 raft_consensus.cc:1261] T 5e8b27dca9af473188c50269820343f1 P b550bed5a64f466c8ba5a1761f3bef39 [term 1 FOLLOWER]: Refusing update from remote peer a64152fd9595478ca767a9f248ef3b85: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:56.636781 28646 raft_consensus.cc:1261] T 5e8b27dca9af473188c50269820343f1 P c541378ca18944c49c37f49097bc7f81 [term 1 FOLLOWER]: Refusing update from remote peer a64152fd9595478ca767a9f248ef3b85: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:56.637322 29961 consensus_queue.cc:1035] T 5e8b27dca9af473188c50269820343f1 P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:56.637650 29832 consensus_queue.cc:1035] T 5e8b27dca9af473188c50269820343f1 P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:56.637996 29829 consensus_queue.cc:1035] T 5e8b27dca9af473188c50269820343f1 P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:56.641337 29832 raft_consensus.cc:2932] T 5e8b27dca9af473188c50269820343f1 P a64152fd9595478ca767a9f248ef3b85 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 543169fd701141c181c55566d2cae00d (127.9.176.196) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } }
I20240502 01:55:56.641944 28778 raft_consensus.cc:2932] T 5e8b27dca9af473188c50269820343f1 P 543169fd701141c181c55566d2cae00d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 543169fd701141c181c55566d2cae00d (127.9.176.196) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } }
I20240502 01:55:56.647823 27983 raft_consensus.cc:2932] T 5e8b27dca9af473188c50269820343f1 P b550bed5a64f466c8ba5a1761f3bef39 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 543169fd701141c181c55566d2cae00d (127.9.176.196) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } }
I20240502 01:55:56.650208 27569 catalog_manager.cc:5484] T 5e8b27dca9af473188c50269820343f1 P 543169fd701141c181c55566d2cae00d reported cstate change: config changed from index 8 to 9, 543169fd701141c181c55566d2cae00d (127.9.176.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a64152fd9595478ca767a9f248ef3b85" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } }
I20240502 01:55:56.653833 28646 raft_consensus.cc:2932] T 5e8b27dca9af473188c50269820343f1 P c541378ca18944c49c37f49097bc7f81 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 543169fd701141c181c55566d2cae00d (127.9.176.196) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } }
I20240502 01:55:56.731478 27849 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5e8b27dca9af473188c50269820343f1"
dest_uuid: "a64152fd9595478ca767a9f248ef3b85"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:53412
I20240502 01:55:56.731775 27849 raft_consensus.cc:595] T 5e8b27dca9af473188c50269820343f1 P a64152fd9595478ca767a9f248ef3b85 [term 1 LEADER]: Received request to transfer leadership
I20240502 01:55:56.742691 29961 raft_consensus.cc:979] T 8a29996e3d594ebebae8b5fe7e02aa8c P a64152fd9595478ca767a9f248ef3b85: : Instructing follower d50f07af87d74f039fb2b43cf166fa43 to start an election
I20240502 01:55:56.742884 29961 raft_consensus.cc:1067] T 8a29996e3d594ebebae8b5fe7e02aa8c P a64152fd9595478ca767a9f248ef3b85 [term 1 LEADER]: Signalling peer d50f07af87d74f039fb2b43cf166fa43 to start an election
I20240502 01:55:56.743641 27715 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8a29996e3d594ebebae8b5fe7e02aa8c"
dest_uuid: "d50f07af87d74f039fb2b43cf166fa43"
 from {username='slave'} at 127.9.176.194:39981
I20240502 01:55:56.743911 27715 raft_consensus.cc:482] T 8a29996e3d594ebebae8b5fe7e02aa8c P d50f07af87d74f039fb2b43cf166fa43 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:55:56.744055 27715 raft_consensus.cc:3037] T 8a29996e3d594ebebae8b5fe7e02aa8c P d50f07af87d74f039fb2b43cf166fa43 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:55:56.746734 27715 raft_consensus.cc:504] T 8a29996e3d594ebebae8b5fe7e02aa8c P d50f07af87d74f039fb2b43cf166fa43 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } }
I20240502 01:55:56.747764 27715 leader_election.cc:288] T 8a29996e3d594ebebae8b5fe7e02aa8c P d50f07af87d74f039fb2b43cf166fa43 [CANDIDATE]: Term 2 election: Requested vote from peers a64152fd9595478ca767a9f248ef3b85 (127.9.176.194:38689), b550bed5a64f466c8ba5a1761f3bef39 (127.9.176.195:40013), c541378ca18944c49c37f49097bc7f81 (127.9.176.197:32773)
I20240502 01:55:56.748093 27849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a29996e3d594ebebae8b5fe7e02aa8c" candidate_uuid: "d50f07af87d74f039fb2b43cf166fa43" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "a64152fd9595478ca767a9f248ef3b85"
I20240502 01:55:56.748342 27849 raft_consensus.cc:3032] T 8a29996e3d594ebebae8b5fe7e02aa8c P a64152fd9595478ca767a9f248ef3b85 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:55:56.748430 27849 raft_consensus.cc:729] T 8a29996e3d594ebebae8b5fe7e02aa8c P a64152fd9595478ca767a9f248ef3b85 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a64152fd9595478ca767a9f248ef3b85, State: Running, Role: LEADER
I20240502 01:55:56.748306 27983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a29996e3d594ebebae8b5fe7e02aa8c" candidate_uuid: "d50f07af87d74f039fb2b43cf166fa43" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "b550bed5a64f466c8ba5a1761f3bef39"
I20240502 01:55:56.748653 27983 raft_consensus.cc:3037] T 8a29996e3d594ebebae8b5fe7e02aa8c P b550bed5a64f466c8ba5a1761f3bef39 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:55:56.748767 27849 consensus_queue.cc:260] T 8a29996e3d594ebebae8b5fe7e02aa8c P a64152fd9595478ca767a9f248ef3b85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } }
I20240502 01:55:56.749197 27849 raft_consensus.cc:3037] T 8a29996e3d594ebebae8b5fe7e02aa8c P a64152fd9595478ca767a9f248ef3b85 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:55:56.751175 27849 raft_consensus.cc:2454] T 8a29996e3d594ebebae8b5fe7e02aa8c P a64152fd9595478ca767a9f248ef3b85 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d50f07af87d74f039fb2b43cf166fa43 in term 2.
I20240502 01:55:56.752374 27983 raft_consensus.cc:2454] T 8a29996e3d594ebebae8b5fe7e02aa8c P b550bed5a64f466c8ba5a1761f3bef39 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d50f07af87d74f039fb2b43cf166fa43 in term 2.
I20240502 01:55:56.753338 27651 leader_election.cc:302] T 8a29996e3d594ebebae8b5fe7e02aa8c P d50f07af87d74f039fb2b43cf166fa43 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: a64152fd9595478ca767a9f248ef3b85, b550bed5a64f466c8ba5a1761f3bef39, d50f07af87d74f039fb2b43cf166fa43; no voters: 
I20240502 01:55:56.753670 29237 raft_consensus.cc:2781] T 8a29996e3d594ebebae8b5fe7e02aa8c P d50f07af87d74f039fb2b43cf166fa43 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:55:56.754627 29237 raft_consensus.cc:686] T 8a29996e3d594ebebae8b5fe7e02aa8c P d50f07af87d74f039fb2b43cf166fa43 [term 2 LEADER]: Becoming Leader. State: Replica: d50f07af87d74f039fb2b43cf166fa43, State: Running, Role: LEADER
I20240502 01:55:56.755113 29237 consensus_queue.cc:237] T 8a29996e3d594ebebae8b5fe7e02aa8c P d50f07af87d74f039fb2b43cf166fa43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } }
I20240502 01:55:56.758543 27566 catalog_manager.cc:5484] T 8a29996e3d594ebebae8b5fe7e02aa8c P d50f07af87d74f039fb2b43cf166fa43 reported cstate change: term changed from 1 to 2, leader changed from a64152fd9595478ca767a9f248ef3b85 (127.9.176.194) to d50f07af87d74f039fb2b43cf166fa43 (127.9.176.193). New cstate: current_term: 2 leader_uuid: "d50f07af87d74f039fb2b43cf166fa43" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:55:56.760727 28646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a29996e3d594ebebae8b5fe7e02aa8c" candidate_uuid: "d50f07af87d74f039fb2b43cf166fa43" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "c541378ca18944c49c37f49097bc7f81"
I20240502 01:55:56.760921 28646 raft_consensus.cc:3037] T 8a29996e3d594ebebae8b5fe7e02aa8c P c541378ca18944c49c37f49097bc7f81 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:55:56.763080 28646 raft_consensus.cc:2454] T 8a29996e3d594ebebae8b5fe7e02aa8c P c541378ca18944c49c37f49097bc7f81 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d50f07af87d74f039fb2b43cf166fa43 in term 2.
I20240502 01:55:56.766315 27849 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8a29996e3d594ebebae8b5fe7e02aa8c"
dest_uuid: "a64152fd9595478ca767a9f248ef3b85"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:53432
I20240502 01:55:56.766542 27849 raft_consensus.cc:595] T 8a29996e3d594ebebae8b5fe7e02aa8c P a64152fd9595478ca767a9f248ef3b85 [term 2 FOLLOWER]: Received request to transfer leadership
I20240502 01:55:56.766638 27849 raft_consensus.cc:603] T 8a29996e3d594ebebae8b5fe7e02aa8c P a64152fd9595478ca767a9f248ef3b85 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240502 01:55:56.995429 27849 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5e8b27dca9af473188c50269820343f1"
dest_uuid: "a64152fd9595478ca767a9f248ef3b85"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:53440
I20240502 01:55:56.995678 27849 raft_consensus.cc:595] T 5e8b27dca9af473188c50269820343f1 P a64152fd9595478ca767a9f248ef3b85 [term 1 LEADER]: Received request to transfer leadership
I20240502 01:55:57.023152 27715 raft_consensus.cc:1261] T 63136d6964084523bb950cc6568a273b P d50f07af87d74f039fb2b43cf166fa43 [term 2 FOLLOWER]: Refusing update from remote peer c541378ca18944c49c37f49097bc7f81: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 01:55:57.023806 29755 consensus_queue.cc:1035] T 63136d6964084523bb950cc6568a273b P c541378ca18944c49c37f49097bc7f81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:57.157191 29832 raft_consensus.cc:979] T 5e8b27dca9af473188c50269820343f1 P a64152fd9595478ca767a9f248ef3b85: : Instructing follower c541378ca18944c49c37f49097bc7f81 to start an election
I20240502 01:55:57.157408 29832 raft_consensus.cc:1067] T 5e8b27dca9af473188c50269820343f1 P a64152fd9595478ca767a9f248ef3b85 [term 1 LEADER]: Signalling peer c541378ca18944c49c37f49097bc7f81 to start an election
I20240502 01:55:57.158011 28646 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5e8b27dca9af473188c50269820343f1"
dest_uuid: "c541378ca18944c49c37f49097bc7f81"
 from {username='slave'} at 127.9.176.194:59883
I20240502 01:55:57.158223 28646 raft_consensus.cc:482] T 5e8b27dca9af473188c50269820343f1 P c541378ca18944c49c37f49097bc7f81 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:55:57.158353 28646 raft_consensus.cc:3037] T 5e8b27dca9af473188c50269820343f1 P c541378ca18944c49c37f49097bc7f81 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:55:57.160260 28646 raft_consensus.cc:504] T 5e8b27dca9af473188c50269820343f1 P c541378ca18944c49c37f49097bc7f81 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } }
I20240502 01:55:57.160902 28646 leader_election.cc:288] T 5e8b27dca9af473188c50269820343f1 P c541378ca18944c49c37f49097bc7f81 [CANDIDATE]: Term 2 election: Requested vote from peers a64152fd9595478ca767a9f248ef3b85 (127.9.176.194:38689), b550bed5a64f466c8ba5a1761f3bef39 (127.9.176.195:40013), 543169fd701141c181c55566d2cae00d (127.9.176.196:38773)
I20240502 01:55:57.161465 28778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e8b27dca9af473188c50269820343f1" candidate_uuid: "c541378ca18944c49c37f49097bc7f81" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "543169fd701141c181c55566d2cae00d"
I20240502 01:55:57.161692 28778 raft_consensus.cc:3037] T 5e8b27dca9af473188c50269820343f1 P 543169fd701141c181c55566d2cae00d [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:55:57.161746 27849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e8b27dca9af473188c50269820343f1" candidate_uuid: "c541378ca18944c49c37f49097bc7f81" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "a64152fd9595478ca767a9f248ef3b85"
I20240502 01:55:57.161764 27983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e8b27dca9af473188c50269820343f1" candidate_uuid: "c541378ca18944c49c37f49097bc7f81" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "b550bed5a64f466c8ba5a1761f3bef39"
I20240502 01:55:57.161952 27849 raft_consensus.cc:3032] T 5e8b27dca9af473188c50269820343f1 P a64152fd9595478ca767a9f248ef3b85 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:55:57.162003 27983 raft_consensus.cc:3037] T 5e8b27dca9af473188c50269820343f1 P b550bed5a64f466c8ba5a1761f3bef39 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:55:57.162034 27849 raft_consensus.cc:729] T 5e8b27dca9af473188c50269820343f1 P a64152fd9595478ca767a9f248ef3b85 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a64152fd9595478ca767a9f248ef3b85, State: Running, Role: LEADER
I20240502 01:55:57.162294 27849 consensus_queue.cc:260] T 5e8b27dca9af473188c50269820343f1 P a64152fd9595478ca767a9f248ef3b85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } }
I20240502 01:55:57.162753 27849 raft_consensus.cc:3037] T 5e8b27dca9af473188c50269820343f1 P a64152fd9595478ca767a9f248ef3b85 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:55:57.164707 28778 raft_consensus.cc:2454] T 5e8b27dca9af473188c50269820343f1 P 543169fd701141c181c55566d2cae00d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c541378ca18944c49c37f49097bc7f81 in term 2.
I20240502 01:55:57.164796 27983 raft_consensus.cc:2454] T 5e8b27dca9af473188c50269820343f1 P b550bed5a64f466c8ba5a1761f3bef39 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c541378ca18944c49c37f49097bc7f81 in term 2.
I20240502 01:55:57.165108 27849 raft_consensus.cc:2454] T 5e8b27dca9af473188c50269820343f1 P a64152fd9595478ca767a9f248ef3b85 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c541378ca18944c49c37f49097bc7f81 in term 2.
I20240502 01:55:57.165305 28580 leader_election.cc:302] T 5e8b27dca9af473188c50269820343f1 P c541378ca18944c49c37f49097bc7f81 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 543169fd701141c181c55566d2cae00d, b550bed5a64f466c8ba5a1761f3bef39, c541378ca18944c49c37f49097bc7f81; no voters: 
I20240502 01:55:57.165601 29755 raft_consensus.cc:2781] T 5e8b27dca9af473188c50269820343f1 P c541378ca18944c49c37f49097bc7f81 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:55:57.165977 29755 raft_consensus.cc:686] T 5e8b27dca9af473188c50269820343f1 P c541378ca18944c49c37f49097bc7f81 [term 2 LEADER]: Becoming Leader. State: Replica: c541378ca18944c49c37f49097bc7f81, State: Running, Role: LEADER
I20240502 01:55:57.166263 29755 consensus_queue.cc:237] T 5e8b27dca9af473188c50269820343f1 P c541378ca18944c49c37f49097bc7f81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } }
I20240502 01:55:57.168782 27566 catalog_manager.cc:5484] T 5e8b27dca9af473188c50269820343f1 P c541378ca18944c49c37f49097bc7f81 reported cstate change: term changed from 1 to 2, leader changed from a64152fd9595478ca767a9f248ef3b85 (127.9.176.194) to c541378ca18944c49c37f49097bc7f81 (127.9.176.197). New cstate: current_term: 2 leader_uuid: "c541378ca18944c49c37f49097bc7f81" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:55:57.173234 27983 raft_consensus.cc:1261] T 8a29996e3d594ebebae8b5fe7e02aa8c P b550bed5a64f466c8ba5a1761f3bef39 [term 2 FOLLOWER]: Refusing update from remote peer d50f07af87d74f039fb2b43cf166fa43: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 01:55:57.173810 29237 consensus_queue.cc:1035] T 8a29996e3d594ebebae8b5fe7e02aa8c P d50f07af87d74f039fb2b43cf166fa43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:57.198745 27983 raft_consensus.cc:1261] T 63136d6964084523bb950cc6568a273b P b550bed5a64f466c8ba5a1761f3bef39 [term 2 FOLLOWER]: Refusing update from remote peer c541378ca18944c49c37f49097bc7f81: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 01:55:57.199424 29755 consensus_queue.cc:1035] T 63136d6964084523bb950cc6568a273b P c541378ca18944c49c37f49097bc7f81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:57.204327 27849 raft_consensus.cc:1261] T 63136d6964084523bb950cc6568a273b P a64152fd9595478ca767a9f248ef3b85 [term 2 FOLLOWER]: Refusing update from remote peer c541378ca18944c49c37f49097bc7f81: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 01:55:57.205181 30075 consensus_queue.cc:1035] T 63136d6964084523bb950cc6568a273b P c541378ca18944c49c37f49097bc7f81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:57.207168 28646 consensus_queue.cc:237] T 63136d6964084523bb950cc6568a273b P c541378ca18944c49c37f49097bc7f81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } }
I20240502 01:55:57.209211 27983 raft_consensus.cc:1261] T 63136d6964084523bb950cc6568a273b P b550bed5a64f466c8ba5a1761f3bef39 [term 2 FOLLOWER]: Refusing update from remote peer c541378ca18944c49c37f49097bc7f81: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 01:55:57.209502 27715 raft_consensus.cc:1261] T 63136d6964084523bb950cc6568a273b P d50f07af87d74f039fb2b43cf166fa43 [term 2 FOLLOWER]: Refusing update from remote peer c541378ca18944c49c37f49097bc7f81: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 01:55:57.210188 30072 consensus_queue.cc:1035] T 63136d6964084523bb950cc6568a273b P c541378ca18944c49c37f49097bc7f81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:57.210747 30075 consensus_queue.cc:1035] T 63136d6964084523bb950cc6568a273b P c541378ca18944c49c37f49097bc7f81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:57.212841 30072 raft_consensus.cc:2932] T 63136d6964084523bb950cc6568a273b P c541378ca18944c49c37f49097bc7f81 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER a64152fd9595478ca767a9f248ef3b85 (127.9.176.194) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } }
I20240502 01:55:57.213778 27715 raft_consensus.cc:2932] T 63136d6964084523bb950cc6568a273b P d50f07af87d74f039fb2b43cf166fa43 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER a64152fd9595478ca767a9f248ef3b85 (127.9.176.194) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } }
I20240502 01:55:57.216553 27983 raft_consensus.cc:2932] T 63136d6964084523bb950cc6568a273b P b550bed5a64f466c8ba5a1761f3bef39 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER a64152fd9595478ca767a9f248ef3b85 (127.9.176.194) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } }
I20240502 01:55:57.216920 27560 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 63136d6964084523bb950cc6568a273b with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:55:57.219090 27566 catalog_manager.cc:5484] T 63136d6964084523bb950cc6568a273b P c541378ca18944c49c37f49097bc7f81 reported cstate change: config changed from index 6 to 8, VOTER a64152fd9595478ca767a9f248ef3b85 (127.9.176.194) evicted. New cstate: current_term: 2 leader_uuid: "c541378ca18944c49c37f49097bc7f81" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:55:57.223682 27827 tablet_service.cc:1514] Processing DeleteTablet for tablet 63136d6964084523bb950cc6568a273b with delete_type TABLET_DATA_TOMBSTONED (TS a64152fd9595478ca767a9f248ef3b85 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:55806
I20240502 01:55:57.224968 30082 tablet_replica.cc:330] T 63136d6964084523bb950cc6568a273b P a64152fd9595478ca767a9f248ef3b85: stopping tablet replica
I20240502 01:55:57.225210 30082 raft_consensus.cc:2229] T 63136d6964084523bb950cc6568a273b P a64152fd9595478ca767a9f248ef3b85 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:57.225418 30082 raft_consensus.cc:2258] T 63136d6964084523bb950cc6568a273b P a64152fd9595478ca767a9f248ef3b85 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:57.226428 30082 ts_tablet_manager.cc:1875] T 63136d6964084523bb950cc6568a273b P a64152fd9595478ca767a9f248ef3b85: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:57.231657 30082 ts_tablet_manager.cc:1888] T 63136d6964084523bb950cc6568a273b P a64152fd9595478ca767a9f248ef3b85: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.7
I20240502 01:55:57.231783 30082 log.cc:1196] T 63136d6964084523bb950cc6568a273b P a64152fd9595478ca767a9f248ef3b85: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/wal/wals/63136d6964084523bb950cc6568a273b
I20240502 01:55:57.232587 27560 catalog_manager.cc:4830] TS a64152fd9595478ca767a9f248ef3b85 (127.9.176.194:38689): tablet 63136d6964084523bb950cc6568a273b (table rebalance_test_table_1 [id=fdfd057dc2884c01bb7ee4631f5e3566]) successfully deleted
I20240502 01:55:57.239979 27849 raft_consensus.cc:1261] T 8a29996e3d594ebebae8b5fe7e02aa8c P a64152fd9595478ca767a9f248ef3b85 [term 2 FOLLOWER]: Refusing update from remote peer d50f07af87d74f039fb2b43cf166fa43: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 01:55:57.240556 29237 consensus_queue.cc:1035] T 8a29996e3d594ebebae8b5fe7e02aa8c P d50f07af87d74f039fb2b43cf166fa43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:57.245389 28646 raft_consensus.cc:1261] T 8a29996e3d594ebebae8b5fe7e02aa8c P c541378ca18944c49c37f49097bc7f81 [term 2 FOLLOWER]: Refusing update from remote peer d50f07af87d74f039fb2b43cf166fa43: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 01:55:57.246498 30044 consensus_queue.cc:1035] T 8a29996e3d594ebebae8b5fe7e02aa8c P d50f07af87d74f039fb2b43cf166fa43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:57.485255 27715 consensus_queue.cc:237] T f22c0dfa05664a4a9f10aa81424a396e P d50f07af87d74f039fb2b43cf166fa43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } }
I20240502 01:55:57.487375 28778 raft_consensus.cc:1261] T f22c0dfa05664a4a9f10aa81424a396e P 543169fd701141c181c55566d2cae00d [term 1 FOLLOWER]: Refusing update from remote peer d50f07af87d74f039fb2b43cf166fa43: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:57.487392 27849 raft_consensus.cc:1261] T f22c0dfa05664a4a9f10aa81424a396e P a64152fd9595478ca767a9f248ef3b85 [term 1 FOLLOWER]: Refusing update from remote peer d50f07af87d74f039fb2b43cf166fa43: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:57.488009 30044 consensus_queue.cc:1035] T f22c0dfa05664a4a9f10aa81424a396e P d50f07af87d74f039fb2b43cf166fa43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:57.488306 29237 consensus_queue.cc:1035] T f22c0dfa05664a4a9f10aa81424a396e P d50f07af87d74f039fb2b43cf166fa43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:57.491243 30044 raft_consensus.cc:2932] T f22c0dfa05664a4a9f10aa81424a396e P d50f07af87d74f039fb2b43cf166fa43 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } }
I20240502 01:55:57.492055 28778 raft_consensus.cc:2932] T f22c0dfa05664a4a9f10aa81424a396e P 543169fd701141c181c55566d2cae00d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } }
I20240502 01:55:57.492269 27849 raft_consensus.cc:2932] T f22c0dfa05664a4a9f10aa81424a396e P a64152fd9595478ca767a9f248ef3b85 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } }
I20240502 01:55:57.497704 27572 catalog_manager.cc:5484] T f22c0dfa05664a4a9f10aa81424a396e P a64152fd9595478ca767a9f248ef3b85 reported cstate change: config changed from index 6 to 7. New cstate: current_term: 1 leader_uuid: "d50f07af87d74f039fb2b43cf166fa43" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } }
I20240502 01:55:57.503903 27715 consensus_queue.cc:237] T f22c0dfa05664a4a9f10aa81424a396e P d50f07af87d74f039fb2b43cf166fa43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: true } }
I20240502 01:55:57.507223 27849 raft_consensus.cc:1261] T f22c0dfa05664a4a9f10aa81424a396e P a64152fd9595478ca767a9f248ef3b85 [term 1 FOLLOWER]: Refusing update from remote peer d50f07af87d74f039fb2b43cf166fa43: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240502 01:55:57.507256 27651 consensus_peers.cc:485] T f22c0dfa05664a4a9f10aa81424a396e P d50f07af87d74f039fb2b43cf166fa43 -> Peer c541378ca18944c49c37f49097bc7f81 (127.9.176.197:32773): Couldn't send request to peer c541378ca18944c49c37f49097bc7f81. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f22c0dfa05664a4a9f10aa81424a396e. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:57.507270 28778 raft_consensus.cc:1261] T f22c0dfa05664a4a9f10aa81424a396e P 543169fd701141c181c55566d2cae00d [term 1 FOLLOWER]: Refusing update from remote peer d50f07af87d74f039fb2b43cf166fa43: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:57.507828 29237 consensus_queue.cc:1035] T f22c0dfa05664a4a9f10aa81424a396e P d50f07af87d74f039fb2b43cf166fa43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:57.508168 30044 consensus_queue.cc:1035] T f22c0dfa05664a4a9f10aa81424a396e P d50f07af87d74f039fb2b43cf166fa43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:57.512141 29237 raft_consensus.cc:2932] T f22c0dfa05664a4a9f10aa81424a396e P d50f07af87d74f039fb2b43cf166fa43 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER c541378ca18944c49c37f49097bc7f81 (127.9.176.197) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: true } } }
I20240502 01:55:57.513092 28778 raft_consensus.cc:2932] T f22c0dfa05664a4a9f10aa81424a396e P 543169fd701141c181c55566d2cae00d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER c541378ca18944c49c37f49097bc7f81 (127.9.176.197) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: true } } }
I20240502 01:55:57.520076 27559 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet f22c0dfa05664a4a9f10aa81424a396e with cas_config_opid_index 7: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 01:55:57.520129 27569 catalog_manager.cc:5484] T f22c0dfa05664a4a9f10aa81424a396e P 543169fd701141c181c55566d2cae00d reported cstate change: config changed from index 7 to 8, NON_VOTER c541378ca18944c49c37f49097bc7f81 (127.9.176.197) added. New cstate: current_term: 1 leader_uuid: "d50f07af87d74f039fb2b43cf166fa43" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: true } } }
I20240502 01:55:57.521387 27849 raft_consensus.cc:2932] T f22c0dfa05664a4a9f10aa81424a396e P a64152fd9595478ca767a9f248ef3b85 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER c541378ca18944c49c37f49097bc7f81 (127.9.176.197) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: true } } }
I20240502 01:55:57.661633 28778 raft_consensus.cc:1261] T 5e8b27dca9af473188c50269820343f1 P 543169fd701141c181c55566d2cae00d [term 2 FOLLOWER]: Refusing update from remote peer c541378ca18944c49c37f49097bc7f81: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 01:55:57.662288 30075 consensus_queue.cc:1035] T 5e8b27dca9af473188c50269820343f1 P c541378ca18944c49c37f49097bc7f81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:57.708544 27983 raft_consensus.cc:1261] T 5e8b27dca9af473188c50269820343f1 P b550bed5a64f466c8ba5a1761f3bef39 [term 2 FOLLOWER]: Refusing update from remote peer c541378ca18944c49c37f49097bc7f81: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 01:55:57.709216 30075 consensus_queue.cc:1035] T 5e8b27dca9af473188c50269820343f1 P c541378ca18944c49c37f49097bc7f81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:57.713819 27849 raft_consensus.cc:1261] T 5e8b27dca9af473188c50269820343f1 P a64152fd9595478ca767a9f248ef3b85 [term 2 FOLLOWER]: Refusing update from remote peer c541378ca18944c49c37f49097bc7f81: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 01:55:57.714972 30075 consensus_queue.cc:1035] T 5e8b27dca9af473188c50269820343f1 P c541378ca18944c49c37f49097bc7f81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:57.716328 28646 consensus_queue.cc:237] T 5e8b27dca9af473188c50269820343f1 P c541378ca18944c49c37f49097bc7f81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } }
I20240502 01:55:57.718820 27983 raft_consensus.cc:1261] T 5e8b27dca9af473188c50269820343f1 P b550bed5a64f466c8ba5a1761f3bef39 [term 2 FOLLOWER]: Refusing update from remote peer c541378ca18944c49c37f49097bc7f81: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240502 01:55:57.719552 30075 consensus_queue.cc:1035] T 5e8b27dca9af473188c50269820343f1 P c541378ca18944c49c37f49097bc7f81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 01:55:57.720450 28778 raft_consensus.cc:1261] T 5e8b27dca9af473188c50269820343f1 P 543169fd701141c181c55566d2cae00d [term 2 FOLLOWER]: Refusing update from remote peer c541378ca18944c49c37f49097bc7f81: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240502 01:55:57.722101 30080 consensus_queue.cc:1037] T 5e8b27dca9af473188c50269820343f1 P c541378ca18944c49c37f49097bc7f81 [LEADER]: Got LMP mismatch error from peer: Peer: permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 2.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 01:55:57.722591 29755 raft_consensus.cc:2932] T 5e8b27dca9af473188c50269820343f1 P c541378ca18944c49c37f49097bc7f81 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER a64152fd9595478ca767a9f248ef3b85 (127.9.176.194) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } }
I20240502 01:55:57.723553 27983 raft_consensus.cc:2932] T 5e8b27dca9af473188c50269820343f1 P b550bed5a64f466c8ba5a1761f3bef39 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER a64152fd9595478ca767a9f248ef3b85 (127.9.176.194) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } }
I20240502 01:55:57.724215 28778 raft_consensus.cc:2932] T 5e8b27dca9af473188c50269820343f1 P 543169fd701141c181c55566d2cae00d [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER a64152fd9595478ca767a9f248ef3b85 (127.9.176.194) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } }
I20240502 01:55:57.725944 27560 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5e8b27dca9af473188c50269820343f1 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:55:57.727391 27570 catalog_manager.cc:5484] T 5e8b27dca9af473188c50269820343f1 P c541378ca18944c49c37f49097bc7f81 reported cstate change: config changed from index 9 to 11, VOTER a64152fd9595478ca767a9f248ef3b85 (127.9.176.194) evicted. New cstate: current_term: 2 leader_uuid: "c541378ca18944c49c37f49097bc7f81" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:55:57.731405 27827 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e8b27dca9af473188c50269820343f1 with delete_type TABLET_DATA_TOMBSTONED (TS a64152fd9595478ca767a9f248ef3b85 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:55806
I20240502 01:55:57.731851 30082 tablet_replica.cc:330] T 5e8b27dca9af473188c50269820343f1 P a64152fd9595478ca767a9f248ef3b85: stopping tablet replica
I20240502 01:55:57.732084 30082 raft_consensus.cc:2229] T 5e8b27dca9af473188c50269820343f1 P a64152fd9595478ca767a9f248ef3b85 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:57.732318 30082 raft_consensus.cc:2258] T 5e8b27dca9af473188c50269820343f1 P a64152fd9595478ca767a9f248ef3b85 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:57.733180 30082 ts_tablet_manager.cc:1875] T 5e8b27dca9af473188c50269820343f1 P a64152fd9595478ca767a9f248ef3b85: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:57.737372 30082 ts_tablet_manager.cc:1888] T 5e8b27dca9af473188c50269820343f1 P a64152fd9595478ca767a9f248ef3b85: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.10
I20240502 01:55:57.737489 30082 log.cc:1196] T 5e8b27dca9af473188c50269820343f1 P a64152fd9595478ca767a9f248ef3b85: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/wal/wals/5e8b27dca9af473188c50269820343f1
I20240502 01:55:57.738128 27560 catalog_manager.cc:4830] TS a64152fd9595478ca767a9f248ef3b85 (127.9.176.194:38689): tablet 5e8b27dca9af473188c50269820343f1 (table rebalance_test_table_1 [id=fdfd057dc2884c01bb7ee4631f5e3566]) successfully deleted
I20240502 01:55:57.815419 27715 consensus_queue.cc:237] T 8a29996e3d594ebebae8b5fe7e02aa8c P d50f07af87d74f039fb2b43cf166fa43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } }
I20240502 01:55:57.817245 28646 raft_consensus.cc:1261] T 8a29996e3d594ebebae8b5fe7e02aa8c P c541378ca18944c49c37f49097bc7f81 [term 2 FOLLOWER]: Refusing update from remote peer d50f07af87d74f039fb2b43cf166fa43: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 01:55:57.817391 27983 raft_consensus.cc:1261] T 8a29996e3d594ebebae8b5fe7e02aa8c P b550bed5a64f466c8ba5a1761f3bef39 [term 2 FOLLOWER]: Refusing update from remote peer d50f07af87d74f039fb2b43cf166fa43: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 01:55:57.817832 29237 consensus_queue.cc:1035] T 8a29996e3d594ebebae8b5fe7e02aa8c P d50f07af87d74f039fb2b43cf166fa43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:57.818181 30087 consensus_queue.cc:1035] T 8a29996e3d594ebebae8b5fe7e02aa8c P d50f07af87d74f039fb2b43cf166fa43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:57.820587 30044 raft_consensus.cc:2932] T 8a29996e3d594ebebae8b5fe7e02aa8c P d50f07af87d74f039fb2b43cf166fa43 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER a64152fd9595478ca767a9f248ef3b85 (127.9.176.194) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } }
I20240502 01:55:57.821189 28646 raft_consensus.cc:2932] T 8a29996e3d594ebebae8b5fe7e02aa8c P c541378ca18944c49c37f49097bc7f81 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER a64152fd9595478ca767a9f248ef3b85 (127.9.176.194) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } }
I20240502 01:55:57.821270 27983 raft_consensus.cc:2932] T 8a29996e3d594ebebae8b5fe7e02aa8c P b550bed5a64f466c8ba5a1761f3bef39 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER a64152fd9595478ca767a9f248ef3b85 (127.9.176.194) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } }
I20240502 01:55:57.824131 27559 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8a29996e3d594ebebae8b5fe7e02aa8c with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:55:57.825116 27569 catalog_manager.cc:5484] T 8a29996e3d594ebebae8b5fe7e02aa8c P d50f07af87d74f039fb2b43cf166fa43 reported cstate change: config changed from index 6 to 8, VOTER a64152fd9595478ca767a9f248ef3b85 (127.9.176.194) evicted. New cstate: current_term: 2 leader_uuid: "d50f07af87d74f039fb2b43cf166fa43" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:55:57.829241 27827 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a29996e3d594ebebae8b5fe7e02aa8c with delete_type TABLET_DATA_TOMBSTONED (TS a64152fd9595478ca767a9f248ef3b85 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:55806
I20240502 01:55:57.829617 30082 tablet_replica.cc:330] T 8a29996e3d594ebebae8b5fe7e02aa8c P a64152fd9595478ca767a9f248ef3b85: stopping tablet replica
I20240502 01:55:57.829770 30082 raft_consensus.cc:2229] T 8a29996e3d594ebebae8b5fe7e02aa8c P a64152fd9595478ca767a9f248ef3b85 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:57.829931 30082 raft_consensus.cc:2258] T 8a29996e3d594ebebae8b5fe7e02aa8c P a64152fd9595478ca767a9f248ef3b85 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:57.830719 30082 ts_tablet_manager.cc:1875] T 8a29996e3d594ebebae8b5fe7e02aa8c P a64152fd9595478ca767a9f248ef3b85: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:57.835098 30082 ts_tablet_manager.cc:1888] T 8a29996e3d594ebebae8b5fe7e02aa8c P a64152fd9595478ca767a9f248ef3b85: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.7
I20240502 01:55:57.835214 30082 log.cc:1196] T 8a29996e3d594ebebae8b5fe7e02aa8c P a64152fd9595478ca767a9f248ef3b85: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/wal/wals/8a29996e3d594ebebae8b5fe7e02aa8c
I20240502 01:55:57.835955 27560 catalog_manager.cc:4830] TS a64152fd9595478ca767a9f248ef3b85 (127.9.176.194:38689): tablet 8a29996e3d594ebebae8b5fe7e02aa8c (table rebalance_test_table_0 [id=8ef52b859d42498b92b6b2ec0805663b]) successfully deleted
I20240502 01:55:57.946776 30140 ts_tablet_manager.cc:902] T f22c0dfa05664a4a9f10aa81424a396e P c541378ca18944c49c37f49097bc7f81: Initiating tablet copy from peer d50f07af87d74f039fb2b43cf166fa43 (127.9.176.193:34203)
I20240502 01:55:57.948300 30140 tablet_copy_client.cc:308] T f22c0dfa05664a4a9f10aa81424a396e P c541378ca18944c49c37f49097bc7f81: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.193:34203
I20240502 01:55:57.949062 27737 tablet_copy_service.cc:140] P d50f07af87d74f039fb2b43cf166fa43: Received BeginTabletCopySession request for tablet f22c0dfa05664a4a9f10aa81424a396e from peer c541378ca18944c49c37f49097bc7f81 ({username='slave'} at 127.9.176.197:50549)
I20240502 01:55:57.949278 27737 tablet_copy_service.cc:161] P d50f07af87d74f039fb2b43cf166fa43: Beginning new tablet copy session on tablet f22c0dfa05664a4a9f10aa81424a396e from peer c541378ca18944c49c37f49097bc7f81 at {username='slave'} at 127.9.176.197:50549: session id = c541378ca18944c49c37f49097bc7f81-f22c0dfa05664a4a9f10aa81424a396e
I20240502 01:55:57.951617 27737 tablet_copy_source_session.cc:205] T f22c0dfa05664a4a9f10aa81424a396e P d50f07af87d74f039fb2b43cf166fa43: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:57.952930 30140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f22c0dfa05664a4a9f10aa81424a396e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:57.957374 30140 tablet_copy_client.cc:783] T f22c0dfa05664a4a9f10aa81424a396e P c541378ca18944c49c37f49097bc7f81: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:57.957629 30140 tablet_copy_client.cc:647] T f22c0dfa05664a4a9f10aa81424a396e P c541378ca18944c49c37f49097bc7f81: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:57.959367 30140 tablet_copy_client.cc:523] T f22c0dfa05664a4a9f10aa81424a396e P c541378ca18944c49c37f49097bc7f81: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:57.961796 30140 tablet_bootstrap.cc:492] T f22c0dfa05664a4a9f10aa81424a396e P c541378ca18944c49c37f49097bc7f81: Bootstrap starting.
I20240502 01:55:57.995929 30140 tablet_bootstrap.cc:492] T f22c0dfa05664a4a9f10aa81424a396e P c541378ca18944c49c37f49097bc7f81: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:57.996369 30140 tablet_bootstrap.cc:492] T f22c0dfa05664a4a9f10aa81424a396e P c541378ca18944c49c37f49097bc7f81: Bootstrap complete.
I20240502 01:55:57.996644 30140 ts_tablet_manager.cc:1367] T f22c0dfa05664a4a9f10aa81424a396e P c541378ca18944c49c37f49097bc7f81: Time spent bootstrapping tablet: real 0.035s	user 0.028s	sys 0.008s
I20240502 01:55:57.997423 30140 raft_consensus.cc:348] T f22c0dfa05664a4a9f10aa81424a396e P c541378ca18944c49c37f49097bc7f81 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: true } }
I20240502 01:55:57.997699 30140 raft_consensus.cc:729] T f22c0dfa05664a4a9f10aa81424a396e P c541378ca18944c49c37f49097bc7f81 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c541378ca18944c49c37f49097bc7f81, State: Initialized, Role: LEARNER
I20240502 01:55:57.997913 30140 consensus_queue.cc:260] T f22c0dfa05664a4a9f10aa81424a396e P c541378ca18944c49c37f49097bc7f81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: NON_VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: true } }
I20240502 01:55:57.999423 30140 ts_tablet_manager.cc:1398] T f22c0dfa05664a4a9f10aa81424a396e P c541378ca18944c49c37f49097bc7f81: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.001s
I20240502 01:55:58.000308 27737 tablet_copy_service.cc:342] P d50f07af87d74f039fb2b43cf166fa43: Request end of tablet copy session c541378ca18944c49c37f49097bc7f81-f22c0dfa05664a4a9f10aa81424a396e received from {username='slave'} at 127.9.176.197:50549
I20240502 01:55:58.000545 27737 tablet_copy_service.cc:433] P d50f07af87d74f039fb2b43cf166fa43: ending tablet copy session c541378ca18944c49c37f49097bc7f81-f22c0dfa05664a4a9f10aa81424a396e on tablet f22c0dfa05664a4a9f10aa81424a396e with peer c541378ca18944c49c37f49097bc7f81
I20240502 01:55:58.006862 27715 consensus_queue.cc:237] T 9a16d465aa2b426484868a2b212c5d00 P d50f07af87d74f039fb2b43cf166fa43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } }
I20240502 01:55:58.009124 27849 raft_consensus.cc:1261] T 9a16d465aa2b426484868a2b212c5d00 P a64152fd9595478ca767a9f248ef3b85 [term 1 FOLLOWER]: Refusing update from remote peer d50f07af87d74f039fb2b43cf166fa43: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:58.009181 28778 raft_consensus.cc:1261] T 9a16d465aa2b426484868a2b212c5d00 P 543169fd701141c181c55566d2cae00d [term 1 FOLLOWER]: Refusing update from remote peer d50f07af87d74f039fb2b43cf166fa43: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:58.009671 29237 consensus_queue.cc:1035] T 9a16d465aa2b426484868a2b212c5d00 P d50f07af87d74f039fb2b43cf166fa43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:58.009984 30087 consensus_queue.cc:1035] T 9a16d465aa2b426484868a2b212c5d00 P d50f07af87d74f039fb2b43cf166fa43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:58.013129 30087 raft_consensus.cc:2932] T 9a16d465aa2b426484868a2b212c5d00 P d50f07af87d74f039fb2b43cf166fa43 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } }
I20240502 01:55:58.013895 28778 raft_consensus.cc:2932] T 9a16d465aa2b426484868a2b212c5d00 P 543169fd701141c181c55566d2cae00d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } }
I20240502 01:55:58.014653 27849 raft_consensus.cc:2932] T 9a16d465aa2b426484868a2b212c5d00 P a64152fd9595478ca767a9f248ef3b85 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } }
I20240502 01:55:58.018806 27566 catalog_manager.cc:5484] T 9a16d465aa2b426484868a2b212c5d00 P d50f07af87d74f039fb2b43cf166fa43 reported cstate change: config changed from index 6 to 7. New cstate: current_term: 1 leader_uuid: "d50f07af87d74f039fb2b43cf166fa43" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:55:58.025223 27715 consensus_queue.cc:237] T 9a16d465aa2b426484868a2b212c5d00 P d50f07af87d74f039fb2b43cf166fa43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: NON_VOTER last_known_addr { host: "127.9.176.195" port: 40013 } attrs { promote: true } }
I20240502 01:55:58.027582 27849 raft_consensus.cc:1261] T 9a16d465aa2b426484868a2b212c5d00 P a64152fd9595478ca767a9f248ef3b85 [term 1 FOLLOWER]: Refusing update from remote peer d50f07af87d74f039fb2b43cf166fa43: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:58.028024 28778 raft_consensus.cc:1261] T 9a16d465aa2b426484868a2b212c5d00 P 543169fd701141c181c55566d2cae00d [term 1 FOLLOWER]: Refusing update from remote peer d50f07af87d74f039fb2b43cf166fa43: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240502 01:55:58.028045 27651 consensus_peers.cc:485] T 9a16d465aa2b426484868a2b212c5d00 P d50f07af87d74f039fb2b43cf166fa43 -> Peer b550bed5a64f466c8ba5a1761f3bef39 (127.9.176.195:40013): Couldn't send request to peer b550bed5a64f466c8ba5a1761f3bef39. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:58.028298 30044 consensus_queue.cc:1035] T 9a16d465aa2b426484868a2b212c5d00 P d50f07af87d74f039fb2b43cf166fa43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:58.028645 29237 consensus_queue.cc:1035] T 9a16d465aa2b426484868a2b212c5d00 P d50f07af87d74f039fb2b43cf166fa43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:58.031136 29237 raft_consensus.cc:2932] T 9a16d465aa2b426484868a2b212c5d00 P d50f07af87d74f039fb2b43cf166fa43 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER b550bed5a64f466c8ba5a1761f3bef39 (127.9.176.195) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: NON_VOTER last_known_addr { host: "127.9.176.195" port: 40013 } attrs { promote: true } } }
I20240502 01:55:58.032642 28778 raft_consensus.cc:2932] T 9a16d465aa2b426484868a2b212c5d00 P 543169fd701141c181c55566d2cae00d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER b550bed5a64f466c8ba5a1761f3bef39 (127.9.176.195) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: NON_VOTER last_known_addr { host: "127.9.176.195" port: 40013 } attrs { promote: true } } }
I20240502 01:55:58.034631 27559 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9a16d465aa2b426484868a2b212c5d00 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 01:55:58.034941 27849 raft_consensus.cc:2932] T 9a16d465aa2b426484868a2b212c5d00 P a64152fd9595478ca767a9f248ef3b85 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER b550bed5a64f466c8ba5a1761f3bef39 (127.9.176.195) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: NON_VOTER last_known_addr { host: "127.9.176.195" port: 40013 } attrs { promote: true } } }
I20240502 01:55:58.036355 27569 catalog_manager.cc:5484] T 9a16d465aa2b426484868a2b212c5d00 P d50f07af87d74f039fb2b43cf166fa43 reported cstate change: config changed from index 7 to 8, NON_VOTER b550bed5a64f466c8ba5a1761f3bef39 (127.9.176.195) added. New cstate: current_term: 1 leader_uuid: "d50f07af87d74f039fb2b43cf166fa43" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: NON_VOTER last_known_addr { host: "127.9.176.195" port: 40013 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:55:58.273576 27849 consensus_queue.cc:237] T b89e275e1e274042a2cc120111080433 P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } }
I20240502 01:55:58.275961 27983 raft_consensus.cc:1261] T b89e275e1e274042a2cc120111080433 P b550bed5a64f466c8ba5a1761f3bef39 [term 1 FOLLOWER]: Refusing update from remote peer a64152fd9595478ca767a9f248ef3b85: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:58.276163 27715 raft_consensus.cc:1261] T b89e275e1e274042a2cc120111080433 P d50f07af87d74f039fb2b43cf166fa43 [term 1 FOLLOWER]: Refusing update from remote peer a64152fd9595478ca767a9f248ef3b85: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:58.276724 29832 consensus_queue.cc:1035] T b89e275e1e274042a2cc120111080433 P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:58.277071 30176 consensus_queue.cc:1035] T b89e275e1e274042a2cc120111080433 P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:58.280166 30176 raft_consensus.cc:2932] T b89e275e1e274042a2cc120111080433 P a64152fd9595478ca767a9f248ef3b85 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } }
I20240502 01:55:58.280934 27715 raft_consensus.cc:2932] T b89e275e1e274042a2cc120111080433 P d50f07af87d74f039fb2b43cf166fa43 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } }
I20240502 01:55:58.281594 27983 raft_consensus.cc:2932] T b89e275e1e274042a2cc120111080433 P b550bed5a64f466c8ba5a1761f3bef39 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } }
I20240502 01:55:58.286216 27566 catalog_manager.cc:5484] T b89e275e1e274042a2cc120111080433 P d50f07af87d74f039fb2b43cf166fa43 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "a64152fd9595478ca767a9f248ef3b85" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } }
I20240502 01:55:58.292090 27849 consensus_queue.cc:237] T b89e275e1e274042a2cc120111080433 P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: true } }
I20240502 01:55:58.294905 27715 raft_consensus.cc:1261] T b89e275e1e274042a2cc120111080433 P d50f07af87d74f039fb2b43cf166fa43 [term 1 FOLLOWER]: Refusing update from remote peer a64152fd9595478ca767a9f248ef3b85: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240502 01:55:58.295516 27782 consensus_peers.cc:485] T b89e275e1e274042a2cc120111080433 P a64152fd9595478ca767a9f248ef3b85 -> Peer 543169fd701141c181c55566d2cae00d (127.9.176.196:38773): Couldn't send request to peer 543169fd701141c181c55566d2cae00d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b89e275e1e274042a2cc120111080433. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:58.295632 29832 consensus_queue.cc:1035] T b89e275e1e274042a2cc120111080433 P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:58.296676 27983 raft_consensus.cc:1261] T b89e275e1e274042a2cc120111080433 P b550bed5a64f466c8ba5a1761f3bef39 [term 1 FOLLOWER]: Refusing update from remote peer a64152fd9595478ca767a9f248ef3b85: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:58.297456 29832 consensus_queue.cc:1035] T b89e275e1e274042a2cc120111080433 P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:58.300228 30176 raft_consensus.cc:2932] T b89e275e1e274042a2cc120111080433 P a64152fd9595478ca767a9f248ef3b85 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 543169fd701141c181c55566d2cae00d (127.9.176.196) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: true } } }
I20240502 01:55:58.301077 27715 raft_consensus.cc:2932] T b89e275e1e274042a2cc120111080433 P d50f07af87d74f039fb2b43cf166fa43 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 543169fd701141c181c55566d2cae00d (127.9.176.196) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: true } } }
I20240502 01:55:58.302050 27981 raft_consensus.cc:2932] T b89e275e1e274042a2cc120111080433 P b550bed5a64f466c8ba5a1761f3bef39 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 543169fd701141c181c55566d2cae00d (127.9.176.196) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: true } } }
I20240502 01:55:58.305227 27560 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet b89e275e1e274042a2cc120111080433 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 01:55:58.307375 27569 catalog_manager.cc:5484] T b89e275e1e274042a2cc120111080433 P d50f07af87d74f039fb2b43cf166fa43 reported cstate change: config changed from index 4 to 5, NON_VOTER 543169fd701141c181c55566d2cae00d (127.9.176.196) added. New cstate: current_term: 1 leader_uuid: "a64152fd9595478ca767a9f248ef3b85" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: true } } }
I20240502 01:55:58.475219 28646 raft_consensus.cc:1203] T f22c0dfa05664a4a9f10aa81424a396e P c541378ca18944c49c37f49097bc7f81 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8]   Dedup: 1.8->[]
I20240502 01:55:58.639122 30210 ts_tablet_manager.cc:902] T 9a16d465aa2b426484868a2b212c5d00 P b550bed5a64f466c8ba5a1761f3bef39: Initiating tablet copy from peer d50f07af87d74f039fb2b43cf166fa43 (127.9.176.193:34203)
I20240502 01:55:58.640816 30210 tablet_copy_client.cc:308] T 9a16d465aa2b426484868a2b212c5d00 P b550bed5a64f466c8ba5a1761f3bef39: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.193:34203
I20240502 01:55:58.641660 27737 tablet_copy_service.cc:140] P d50f07af87d74f039fb2b43cf166fa43: Received BeginTabletCopySession request for tablet 9a16d465aa2b426484868a2b212c5d00 from peer b550bed5a64f466c8ba5a1761f3bef39 ({username='slave'} at 127.9.176.195:41909)
I20240502 01:55:58.641822 27737 tablet_copy_service.cc:161] P d50f07af87d74f039fb2b43cf166fa43: Beginning new tablet copy session on tablet 9a16d465aa2b426484868a2b212c5d00 from peer b550bed5a64f466c8ba5a1761f3bef39 at {username='slave'} at 127.9.176.195:41909: session id = b550bed5a64f466c8ba5a1761f3bef39-9a16d465aa2b426484868a2b212c5d00
I20240502 01:55:58.643841 27737 tablet_copy_source_session.cc:205] T 9a16d465aa2b426484868a2b212c5d00 P d50f07af87d74f039fb2b43cf166fa43: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:58.645354 30210 ts_tablet_manager.cc:1875] T 9a16d465aa2b426484868a2b212c5d00 P b550bed5a64f466c8ba5a1761f3bef39: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 01:55:58.650239 30210 ts_tablet_manager.cc:1888] T 9a16d465aa2b426484868a2b212c5d00 P b550bed5a64f466c8ba5a1761f3bef39: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240502 01:55:58.650532 30210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a16d465aa2b426484868a2b212c5d00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:58.653092 30210 tablet_copy_client.cc:783] T 9a16d465aa2b426484868a2b212c5d00 P b550bed5a64f466c8ba5a1761f3bef39: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:58.653463 30210 tablet_copy_client.cc:647] T 9a16d465aa2b426484868a2b212c5d00 P b550bed5a64f466c8ba5a1761f3bef39: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:58.655825 30210 tablet_copy_client.cc:523] T 9a16d465aa2b426484868a2b212c5d00 P b550bed5a64f466c8ba5a1761f3bef39: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:58.658820 30210 tablet_bootstrap.cc:492] T 9a16d465aa2b426484868a2b212c5d00 P b550bed5a64f466c8ba5a1761f3bef39: Bootstrap starting.
I20240502 01:55:58.694481 30210 tablet_bootstrap.cc:492] T 9a16d465aa2b426484868a2b212c5d00 P b550bed5a64f466c8ba5a1761f3bef39: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:58.694974 30210 tablet_bootstrap.cc:492] T 9a16d465aa2b426484868a2b212c5d00 P b550bed5a64f466c8ba5a1761f3bef39: Bootstrap complete.
I20240502 01:55:58.695294 30210 ts_tablet_manager.cc:1367] T 9a16d465aa2b426484868a2b212c5d00 P b550bed5a64f466c8ba5a1761f3bef39: Time spent bootstrapping tablet: real 0.037s	user 0.031s	sys 0.005s
I20240502 01:55:58.696024 30210 raft_consensus.cc:348] T 9a16d465aa2b426484868a2b212c5d00 P b550bed5a64f466c8ba5a1761f3bef39 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: NON_VOTER last_known_addr { host: "127.9.176.195" port: 40013 } attrs { promote: true } }
I20240502 01:55:58.696269 30210 raft_consensus.cc:729] T 9a16d465aa2b426484868a2b212c5d00 P b550bed5a64f466c8ba5a1761f3bef39 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b550bed5a64f466c8ba5a1761f3bef39, State: Initialized, Role: LEARNER
I20240502 01:55:58.696496 30210 consensus_queue.cc:260] T 9a16d465aa2b426484868a2b212c5d00 P b550bed5a64f466c8ba5a1761f3bef39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: NON_VOTER last_known_addr { host: "127.9.176.195" port: 40013 } attrs { promote: true } }
I20240502 01:55:58.697073 30210 ts_tablet_manager.cc:1398] T 9a16d465aa2b426484868a2b212c5d00 P b550bed5a64f466c8ba5a1761f3bef39: Time spent starting tablet: real 0.002s	user 0.004s	sys 0.000s
I20240502 01:55:58.698302 27737 tablet_copy_service.cc:342] P d50f07af87d74f039fb2b43cf166fa43: Request end of tablet copy session b550bed5a64f466c8ba5a1761f3bef39-9a16d465aa2b426484868a2b212c5d00 received from {username='slave'} at 127.9.176.195:41909
I20240502 01:55:58.698477 27737 tablet_copy_service.cc:433] P d50f07af87d74f039fb2b43cf166fa43: ending tablet copy session b550bed5a64f466c8ba5a1761f3bef39-9a16d465aa2b426484868a2b212c5d00 on tablet 9a16d465aa2b426484868a2b212c5d00 with peer b550bed5a64f466c8ba5a1761f3bef39
I20240502 01:55:58.852339 30231 ts_tablet_manager.cc:902] T b89e275e1e274042a2cc120111080433 P 543169fd701141c181c55566d2cae00d: Initiating tablet copy from peer a64152fd9595478ca767a9f248ef3b85 (127.9.176.194:38689)
I20240502 01:55:58.856657 30231 tablet_copy_client.cc:308] T b89e275e1e274042a2cc120111080433 P 543169fd701141c181c55566d2cae00d: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.194:38689
I20240502 01:55:58.857447 27869 tablet_copy_service.cc:140] P a64152fd9595478ca767a9f248ef3b85: Received BeginTabletCopySession request for tablet b89e275e1e274042a2cc120111080433 from peer 543169fd701141c181c55566d2cae00d ({username='slave'} at 127.9.176.196:38793)
I20240502 01:55:58.857615 27869 tablet_copy_service.cc:161] P a64152fd9595478ca767a9f248ef3b85: Beginning new tablet copy session on tablet b89e275e1e274042a2cc120111080433 from peer 543169fd701141c181c55566d2cae00d at {username='slave'} at 127.9.176.196:38793: session id = 543169fd701141c181c55566d2cae00d-b89e275e1e274042a2cc120111080433
I20240502 01:55:58.859792 27869 tablet_copy_source_session.cc:205] T b89e275e1e274042a2cc120111080433 P a64152fd9595478ca767a9f248ef3b85: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:58.861040 30231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b89e275e1e274042a2cc120111080433. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:58.865689 30231 tablet_copy_client.cc:783] T b89e275e1e274042a2cc120111080433 P 543169fd701141c181c55566d2cae00d: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:58.865947 30231 tablet_copy_client.cc:647] T b89e275e1e274042a2cc120111080433 P 543169fd701141c181c55566d2cae00d: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:58.867765 30231 tablet_copy_client.cc:523] T b89e275e1e274042a2cc120111080433 P 543169fd701141c181c55566d2cae00d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:58.870286 30231 tablet_bootstrap.cc:492] T b89e275e1e274042a2cc120111080433 P 543169fd701141c181c55566d2cae00d: Bootstrap starting.
I20240502 01:55:58.905012 30231 tablet_bootstrap.cc:492] T b89e275e1e274042a2cc120111080433 P 543169fd701141c181c55566d2cae00d: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:58.905454 30231 tablet_bootstrap.cc:492] T b89e275e1e274042a2cc120111080433 P 543169fd701141c181c55566d2cae00d: Bootstrap complete.
I20240502 01:55:58.905722 30231 ts_tablet_manager.cc:1367] T b89e275e1e274042a2cc120111080433 P 543169fd701141c181c55566d2cae00d: Time spent bootstrapping tablet: real 0.036s	user 0.031s	sys 0.005s
I20240502 01:55:58.906415 30231 raft_consensus.cc:348] T b89e275e1e274042a2cc120111080433 P 543169fd701141c181c55566d2cae00d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: true } }
I20240502 01:55:58.906631 30231 raft_consensus.cc:729] T b89e275e1e274042a2cc120111080433 P 543169fd701141c181c55566d2cae00d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 543169fd701141c181c55566d2cae00d, State: Initialized, Role: LEARNER
I20240502 01:55:58.906831 30231 consensus_queue.cc:260] T b89e275e1e274042a2cc120111080433 P 543169fd701141c181c55566d2cae00d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: NON_VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: true } }
I20240502 01:55:58.907315 30231 ts_tablet_manager.cc:1398] T b89e275e1e274042a2cc120111080433 P 543169fd701141c181c55566d2cae00d: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240502 01:55:58.908160 27869 tablet_copy_service.cc:342] P a64152fd9595478ca767a9f248ef3b85: Request end of tablet copy session 543169fd701141c181c55566d2cae00d-b89e275e1e274042a2cc120111080433 received from {username='slave'} at 127.9.176.196:38793
I20240502 01:55:58.908319 27869 tablet_copy_service.cc:433] P a64152fd9595478ca767a9f248ef3b85: ending tablet copy session 543169fd701141c181c55566d2cae00d-b89e275e1e274042a2cc120111080433 on tablet b89e275e1e274042a2cc120111080433 with peer 543169fd701141c181c55566d2cae00d
I20240502 01:55:58.929718 30044 raft_consensus.cc:1050] T f22c0dfa05664a4a9f10aa81424a396e P d50f07af87d74f039fb2b43cf166fa43: attempting to promote NON_VOTER c541378ca18944c49c37f49097bc7f81 to VOTER
I20240502 01:55:58.930420 30044 consensus_queue.cc:237] T f22c0dfa05664a4a9f10aa81424a396e P d50f07af87d74f039fb2b43cf166fa43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } }
I20240502 01:55:58.932873 28778 raft_consensus.cc:1261] T f22c0dfa05664a4a9f10aa81424a396e P 543169fd701141c181c55566d2cae00d [term 1 FOLLOWER]: Refusing update from remote peer d50f07af87d74f039fb2b43cf166fa43: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:58.932924 28646 raft_consensus.cc:1261] T f22c0dfa05664a4a9f10aa81424a396e P c541378ca18944c49c37f49097bc7f81 [term 1 LEARNER]: Refusing update from remote peer d50f07af87d74f039fb2b43cf166fa43: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:58.932962 27849 raft_consensus.cc:1261] T f22c0dfa05664a4a9f10aa81424a396e P a64152fd9595478ca767a9f248ef3b85 [term 1 FOLLOWER]: Refusing update from remote peer d50f07af87d74f039fb2b43cf166fa43: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:58.933612 30238 consensus_queue.cc:1035] T f22c0dfa05664a4a9f10aa81424a396e P d50f07af87d74f039fb2b43cf166fa43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:58.934064 30044 consensus_queue.cc:1035] T f22c0dfa05664a4a9f10aa81424a396e P d50f07af87d74f039fb2b43cf166fa43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:58.934377 30087 consensus_queue.cc:1035] T f22c0dfa05664a4a9f10aa81424a396e P d50f07af87d74f039fb2b43cf166fa43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:58.939075 30238 raft_consensus.cc:2932] T f22c0dfa05664a4a9f10aa81424a396e P d50f07af87d74f039fb2b43cf166fa43 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, c541378ca18944c49c37f49097bc7f81 (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } }
I20240502 01:55:58.940022 28646 raft_consensus.cc:2932] T f22c0dfa05664a4a9f10aa81424a396e P c541378ca18944c49c37f49097bc7f81 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, c541378ca18944c49c37f49097bc7f81 (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } }
I20240502 01:55:58.940222 28778 raft_consensus.cc:2932] T f22c0dfa05664a4a9f10aa81424a396e P 543169fd701141c181c55566d2cae00d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, c541378ca18944c49c37f49097bc7f81 (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } }
I20240502 01:55:58.944569 27566 catalog_manager.cc:5484] T f22c0dfa05664a4a9f10aa81424a396e P c541378ca18944c49c37f49097bc7f81 reported cstate change: config changed from index 8 to 9, c541378ca18944c49c37f49097bc7f81 (127.9.176.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d50f07af87d74f039fb2b43cf166fa43" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } }
I20240502 01:55:58.946415 27849 raft_consensus.cc:2932] T f22c0dfa05664a4a9f10aa81424a396e P a64152fd9595478ca767a9f248ef3b85 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, c541378ca18944c49c37f49097bc7f81 (127.9.176.197) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } }
I20240502 01:55:58.950837 27715 consensus_queue.cc:237] T f22c0dfa05664a4a9f10aa81424a396e P d50f07af87d74f039fb2b43cf166fa43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } }
I20240502 01:55:58.953001 28778 raft_consensus.cc:1261] T f22c0dfa05664a4a9f10aa81424a396e P 543169fd701141c181c55566d2cae00d [term 1 FOLLOWER]: Refusing update from remote peer d50f07af87d74f039fb2b43cf166fa43: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:58.953431 28646 raft_consensus.cc:1261] T f22c0dfa05664a4a9f10aa81424a396e P c541378ca18944c49c37f49097bc7f81 [term 1 FOLLOWER]: Refusing update from remote peer d50f07af87d74f039fb2b43cf166fa43: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:58.954012 30238 consensus_queue.cc:1035] T f22c0dfa05664a4a9f10aa81424a396e P d50f07af87d74f039fb2b43cf166fa43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:58.954432 30087 consensus_queue.cc:1035] T f22c0dfa05664a4a9f10aa81424a396e P d50f07af87d74f039fb2b43cf166fa43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:58.957044 30087 raft_consensus.cc:2932] T f22c0dfa05664a4a9f10aa81424a396e P d50f07af87d74f039fb2b43cf166fa43 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER a64152fd9595478ca767a9f248ef3b85 (127.9.176.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } }
I20240502 01:55:58.957784 28646 raft_consensus.cc:2932] T f22c0dfa05664a4a9f10aa81424a396e P c541378ca18944c49c37f49097bc7f81 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER a64152fd9595478ca767a9f248ef3b85 (127.9.176.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } }
I20240502 01:55:58.961081 27559 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f22c0dfa05664a4a9f10aa81424a396e with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:55:58.961709 28778 raft_consensus.cc:2932] T f22c0dfa05664a4a9f10aa81424a396e P 543169fd701141c181c55566d2cae00d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER a64152fd9595478ca767a9f248ef3b85 (127.9.176.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } } }
I20240502 01:55:58.961891 27569 catalog_manager.cc:5484] T f22c0dfa05664a4a9f10aa81424a396e P d50f07af87d74f039fb2b43cf166fa43 reported cstate change: config changed from index 9 to 10, VOTER a64152fd9595478ca767a9f248ef3b85 (127.9.176.194) evicted. New cstate: current_term: 1 leader_uuid: "d50f07af87d74f039fb2b43cf166fa43" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c541378ca18944c49c37f49097bc7f81" member_type: VOTER last_known_addr { host: "127.9.176.197" port: 32773 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:55:58.965857 27827 tablet_service.cc:1514] Processing DeleteTablet for tablet f22c0dfa05664a4a9f10aa81424a396e with delete_type TABLET_DATA_TOMBSTONED (TS a64152fd9595478ca767a9f248ef3b85 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:55806
I20240502 01:55:58.966650 30245 tablet_replica.cc:330] T f22c0dfa05664a4a9f10aa81424a396e P a64152fd9595478ca767a9f248ef3b85: stopping tablet replica
I20240502 01:55:58.967063 30245 raft_consensus.cc:2229] T f22c0dfa05664a4a9f10aa81424a396e P a64152fd9595478ca767a9f248ef3b85 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:58.967483 30245 raft_consensus.cc:2258] T f22c0dfa05664a4a9f10aa81424a396e P a64152fd9595478ca767a9f248ef3b85 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:58.968643 30245 ts_tablet_manager.cc:1875] T f22c0dfa05664a4a9f10aa81424a396e P a64152fd9595478ca767a9f248ef3b85: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:58.973455 30245 ts_tablet_manager.cc:1888] T f22c0dfa05664a4a9f10aa81424a396e P a64152fd9595478ca767a9f248ef3b85: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240502 01:55:58.973595 30245 log.cc:1196] T f22c0dfa05664a4a9f10aa81424a396e P a64152fd9595478ca767a9f248ef3b85: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/wal/wals/f22c0dfa05664a4a9f10aa81424a396e
I20240502 01:55:58.974531 27560 catalog_manager.cc:4830] TS a64152fd9595478ca767a9f248ef3b85 (127.9.176.194:38689): tablet f22c0dfa05664a4a9f10aa81424a396e (table rebalance_test_table_1 [id=fdfd057dc2884c01bb7ee4631f5e3566]) successfully deleted
I20240502 01:55:59.089876 27981 raft_consensus.cc:1203] T 9a16d465aa2b426484868a2b212c5d00 P b550bed5a64f466c8ba5a1761f3bef39 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8]   Dedup: 1.8->[]
I20240502 01:55:59.453079 28778 raft_consensus.cc:1203] T b89e275e1e274042a2cc120111080433 P 543169fd701141c181c55566d2cae00d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
I20240502 01:55:59.637923 30283 raft_consensus.cc:1050] T 9a16d465aa2b426484868a2b212c5d00 P d50f07af87d74f039fb2b43cf166fa43: attempting to promote NON_VOTER b550bed5a64f466c8ba5a1761f3bef39 to VOTER
I20240502 01:55:59.638630 30283 consensus_queue.cc:237] T 9a16d465aa2b426484868a2b212c5d00 P d50f07af87d74f039fb2b43cf166fa43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } attrs { promote: false } }
I20240502 01:55:59.640998 27981 raft_consensus.cc:1261] T 9a16d465aa2b426484868a2b212c5d00 P b550bed5a64f466c8ba5a1761f3bef39 [term 1 LEARNER]: Refusing update from remote peer d50f07af87d74f039fb2b43cf166fa43: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:59.641130 28778 raft_consensus.cc:1261] T 9a16d465aa2b426484868a2b212c5d00 P 543169fd701141c181c55566d2cae00d [term 1 FOLLOWER]: Refusing update from remote peer d50f07af87d74f039fb2b43cf166fa43: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:59.641413 27849 raft_consensus.cc:1261] T 9a16d465aa2b426484868a2b212c5d00 P a64152fd9595478ca767a9f248ef3b85 [term 1 FOLLOWER]: Refusing update from remote peer d50f07af87d74f039fb2b43cf166fa43: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:59.641706 30285 consensus_queue.cc:1035] T 9a16d465aa2b426484868a2b212c5d00 P d50f07af87d74f039fb2b43cf166fa43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:59.642069 30283 consensus_queue.cc:1035] T 9a16d465aa2b426484868a2b212c5d00 P d50f07af87d74f039fb2b43cf166fa43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:59.642378 30087 consensus_queue.cc:1035] T 9a16d465aa2b426484868a2b212c5d00 P d50f07af87d74f039fb2b43cf166fa43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:59.649570 30087 raft_consensus.cc:2932] T 9a16d465aa2b426484868a2b212c5d00 P d50f07af87d74f039fb2b43cf166fa43 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b550bed5a64f466c8ba5a1761f3bef39 (127.9.176.195) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } attrs { promote: false } } }
I20240502 01:55:59.650218 27849 raft_consensus.cc:2932] T 9a16d465aa2b426484868a2b212c5d00 P a64152fd9595478ca767a9f248ef3b85 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b550bed5a64f466c8ba5a1761f3bef39 (127.9.176.195) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } attrs { promote: false } } }
I20240502 01:55:59.650247 27981 raft_consensus.cc:2932] T 9a16d465aa2b426484868a2b212c5d00 P b550bed5a64f466c8ba5a1761f3bef39 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b550bed5a64f466c8ba5a1761f3bef39 (127.9.176.195) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } attrs { promote: false } } }
I20240502 01:55:59.655917 28778 raft_consensus.cc:2932] T 9a16d465aa2b426484868a2b212c5d00 P 543169fd701141c181c55566d2cae00d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b550bed5a64f466c8ba5a1761f3bef39 (127.9.176.195) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } attrs { promote: false } } }
I20240502 01:55:59.656651 27569 catalog_manager.cc:5484] T 9a16d465aa2b426484868a2b212c5d00 P d50f07af87d74f039fb2b43cf166fa43 reported cstate change: config changed from index 8 to 9, b550bed5a64f466c8ba5a1761f3bef39 (127.9.176.195) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d50f07af87d74f039fb2b43cf166fa43" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:55:59.665357 27715 consensus_queue.cc:237] T 9a16d465aa2b426484868a2b212c5d00 P d50f07af87d74f039fb2b43cf166fa43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } attrs { promote: false } }
I20240502 01:55:59.667245 27981 raft_consensus.cc:1261] T 9a16d465aa2b426484868a2b212c5d00 P b550bed5a64f466c8ba5a1761f3bef39 [term 1 FOLLOWER]: Refusing update from remote peer d50f07af87d74f039fb2b43cf166fa43: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:59.667259 28778 raft_consensus.cc:1261] T 9a16d465aa2b426484868a2b212c5d00 P 543169fd701141c181c55566d2cae00d [term 1 FOLLOWER]: Refusing update from remote peer d50f07af87d74f039fb2b43cf166fa43: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:59.667830 30283 consensus_queue.cc:1035] T 9a16d465aa2b426484868a2b212c5d00 P d50f07af87d74f039fb2b43cf166fa43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:59.668148 30285 consensus_queue.cc:1035] T 9a16d465aa2b426484868a2b212c5d00 P d50f07af87d74f039fb2b43cf166fa43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:59.671021 30283 raft_consensus.cc:2932] T 9a16d465aa2b426484868a2b212c5d00 P d50f07af87d74f039fb2b43cf166fa43 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER a64152fd9595478ca767a9f248ef3b85 (127.9.176.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } attrs { promote: false } } }
I20240502 01:55:59.672302 27981 raft_consensus.cc:2932] T 9a16d465aa2b426484868a2b212c5d00 P b550bed5a64f466c8ba5a1761f3bef39 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER a64152fd9595478ca767a9f248ef3b85 (127.9.176.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } attrs { promote: false } } }
I20240502 01:55:59.675797 27559 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9a16d465aa2b426484868a2b212c5d00 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:55:59.676539 27570 catalog_manager.cc:5484] T 9a16d465aa2b426484868a2b212c5d00 P d50f07af87d74f039fb2b43cf166fa43 reported cstate change: config changed from index 9 to 10, VOTER a64152fd9595478ca767a9f248ef3b85 (127.9.176.194) evicted. New cstate: current_term: 1 leader_uuid: "d50f07af87d74f039fb2b43cf166fa43" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:55:59.677016 28778 raft_consensus.cc:2932] T 9a16d465aa2b426484868a2b212c5d00 P 543169fd701141c181c55566d2cae00d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER a64152fd9595478ca767a9f248ef3b85 (127.9.176.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } attrs { promote: false } } }
I20240502 01:55:59.681913 27827 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a16d465aa2b426484868a2b212c5d00 with delete_type TABLET_DATA_TOMBSTONED (TS a64152fd9595478ca767a9f248ef3b85 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:55806
I20240502 01:55:59.688202 30298 tablet_replica.cc:330] T 9a16d465aa2b426484868a2b212c5d00 P a64152fd9595478ca767a9f248ef3b85: stopping tablet replica
I20240502 01:55:59.688508 30298 raft_consensus.cc:2229] T 9a16d465aa2b426484868a2b212c5d00 P a64152fd9595478ca767a9f248ef3b85 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:59.689028 30298 raft_consensus.cc:2258] T 9a16d465aa2b426484868a2b212c5d00 P a64152fd9595478ca767a9f248ef3b85 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:59.690033 30298 ts_tablet_manager.cc:1875] T 9a16d465aa2b426484868a2b212c5d00 P a64152fd9595478ca767a9f248ef3b85: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:59.696134 30298 ts_tablet_manager.cc:1888] T 9a16d465aa2b426484868a2b212c5d00 P a64152fd9595478ca767a9f248ef3b85: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240502 01:55:59.696323 30298 log.cc:1196] T 9a16d465aa2b426484868a2b212c5d00 P a64152fd9595478ca767a9f248ef3b85: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/wal/wals/9a16d465aa2b426484868a2b212c5d00
I20240502 01:55:59.697345 27560 catalog_manager.cc:4830] TS a64152fd9595478ca767a9f248ef3b85 (127.9.176.194:38689): tablet 9a16d465aa2b426484868a2b212c5d00 (table rebalance_test_table_0 [id=8ef52b859d42498b92b6b2ec0805663b]) successfully deleted
I20240502 01:55:59.908128 30277 raft_consensus.cc:1050] T b89e275e1e274042a2cc120111080433 P a64152fd9595478ca767a9f248ef3b85: attempting to promote NON_VOTER 543169fd701141c181c55566d2cae00d to VOTER
I20240502 01:55:59.909085 30277 consensus_queue.cc:237] T b89e275e1e274042a2cc120111080433 P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } }
I20240502 01:55:59.912144 27715 raft_consensus.cc:1261] T b89e275e1e274042a2cc120111080433 P d50f07af87d74f039fb2b43cf166fa43 [term 1 FOLLOWER]: Refusing update from remote peer a64152fd9595478ca767a9f248ef3b85: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:59.912257 28778 raft_consensus.cc:1261] T b89e275e1e274042a2cc120111080433 P 543169fd701141c181c55566d2cae00d [term 1 LEARNER]: Refusing update from remote peer a64152fd9595478ca767a9f248ef3b85: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:59.912864 27981 raft_consensus.cc:1261] T b89e275e1e274042a2cc120111080433 P b550bed5a64f466c8ba5a1761f3bef39 [term 1 FOLLOWER]: Refusing update from remote peer a64152fd9595478ca767a9f248ef3b85: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:59.913089 30310 consensus_queue.cc:1035] T b89e275e1e274042a2cc120111080433 P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:59.913949 30310 consensus_queue.cc:1035] T b89e275e1e274042a2cc120111080433 P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:59.914310 30277 consensus_queue.cc:1035] T b89e275e1e274042a2cc120111080433 P a64152fd9595478ca767a9f248ef3b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:59.917379 30277 raft_consensus.cc:2932] T b89e275e1e274042a2cc120111080433 P a64152fd9595478ca767a9f248ef3b85 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 543169fd701141c181c55566d2cae00d (127.9.176.196) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } }
I20240502 01:55:59.918243 28778 raft_consensus.cc:2932] T b89e275e1e274042a2cc120111080433 P 543169fd701141c181c55566d2cae00d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 543169fd701141c181c55566d2cae00d (127.9.176.196) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } }
I20240502 01:55:59.921355 27715 raft_consensus.cc:2932] T b89e275e1e274042a2cc120111080433 P d50f07af87d74f039fb2b43cf166fa43 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 543169fd701141c181c55566d2cae00d (127.9.176.196) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } }
I20240502 01:55:59.922271 27570 catalog_manager.cc:5484] T b89e275e1e274042a2cc120111080433 P a64152fd9595478ca767a9f248ef3b85 reported cstate change: config changed from index 5 to 6, 543169fd701141c181c55566d2cae00d (127.9.176.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a64152fd9595478ca767a9f248ef3b85" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:55:59.925288 27981 raft_consensus.cc:2932] T b89e275e1e274042a2cc120111080433 P b550bed5a64f466c8ba5a1761f3bef39 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 543169fd701141c181c55566d2cae00d (127.9.176.196) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } }
I20240502 01:56:00.129771 27849 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b89e275e1e274042a2cc120111080433"
dest_uuid: "a64152fd9595478ca767a9f248ef3b85"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:59190
I20240502 01:56:00.130012 27849 raft_consensus.cc:595] T b89e275e1e274042a2cc120111080433 P a64152fd9595478ca767a9f248ef3b85 [term 1 LEADER]: Received request to transfer leadership
I20240502 01:56:00.348415 27849 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b89e275e1e274042a2cc120111080433"
dest_uuid: "a64152fd9595478ca767a9f248ef3b85"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:59210
I20240502 01:56:00.348683 27849 raft_consensus.cc:595] T b89e275e1e274042a2cc120111080433 P a64152fd9595478ca767a9f248ef3b85 [term 1 LEADER]: Received request to transfer leadership
I20240502 01:56:00.452386 30335 raft_consensus.cc:979] T b89e275e1e274042a2cc120111080433 P a64152fd9595478ca767a9f248ef3b85: : Instructing follower 543169fd701141c181c55566d2cae00d to start an election
I20240502 01:56:00.452572 30335 raft_consensus.cc:1067] T b89e275e1e274042a2cc120111080433 P a64152fd9595478ca767a9f248ef3b85 [term 1 LEADER]: Signalling peer 543169fd701141c181c55566d2cae00d to start an election
I20240502 01:56:00.453227 28778 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b89e275e1e274042a2cc120111080433"
dest_uuid: "543169fd701141c181c55566d2cae00d"
 from {username='slave'} at 127.9.176.194:36721
I20240502 01:56:00.453444 28778 raft_consensus.cc:482] T b89e275e1e274042a2cc120111080433 P 543169fd701141c181c55566d2cae00d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:56:00.453586 28778 raft_consensus.cc:3037] T b89e275e1e274042a2cc120111080433 P 543169fd701141c181c55566d2cae00d [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:56:00.455585 28778 raft_consensus.cc:504] T b89e275e1e274042a2cc120111080433 P 543169fd701141c181c55566d2cae00d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } }
I20240502 01:56:00.456468 28778 leader_election.cc:288] T b89e275e1e274042a2cc120111080433 P 543169fd701141c181c55566d2cae00d [CANDIDATE]: Term 2 election: Requested vote from peers d50f07af87d74f039fb2b43cf166fa43 (127.9.176.193:34203), a64152fd9595478ca767a9f248ef3b85 (127.9.176.194:38689), b550bed5a64f466c8ba5a1761f3bef39 (127.9.176.195:40013)
I20240502 01:56:00.456938 27715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b89e275e1e274042a2cc120111080433" candidate_uuid: "543169fd701141c181c55566d2cae00d" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "d50f07af87d74f039fb2b43cf166fa43"
I20240502 01:56:00.456967 27849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b89e275e1e274042a2cc120111080433" candidate_uuid: "543169fd701141c181c55566d2cae00d" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "a64152fd9595478ca767a9f248ef3b85"
I20240502 01:56:00.457093 27981 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b89e275e1e274042a2cc120111080433" candidate_uuid: "543169fd701141c181c55566d2cae00d" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "b550bed5a64f466c8ba5a1761f3bef39"
I20240502 01:56:00.457191 27715 raft_consensus.cc:3037] T b89e275e1e274042a2cc120111080433 P d50f07af87d74f039fb2b43cf166fa43 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:56:00.457207 27849 raft_consensus.cc:3032] T b89e275e1e274042a2cc120111080433 P a64152fd9595478ca767a9f248ef3b85 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:56:00.457290 27849 raft_consensus.cc:729] T b89e275e1e274042a2cc120111080433 P a64152fd9595478ca767a9f248ef3b85 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a64152fd9595478ca767a9f248ef3b85, State: Running, Role: LEADER
I20240502 01:56:00.457348 27981 raft_consensus.cc:3037] T b89e275e1e274042a2cc120111080433 P b550bed5a64f466c8ba5a1761f3bef39 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:56:00.457643 27849 consensus_queue.cc:260] T b89e275e1e274042a2cc120111080433 P a64152fd9595478ca767a9f248ef3b85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } }
I20240502 01:56:00.458112 27849 raft_consensus.cc:3037] T b89e275e1e274042a2cc120111080433 P a64152fd9595478ca767a9f248ef3b85 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:56:00.459146 27715 raft_consensus.cc:2454] T b89e275e1e274042a2cc120111080433 P d50f07af87d74f039fb2b43cf166fa43 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 543169fd701141c181c55566d2cae00d in term 2.
I20240502 01:56:00.460247 27981 raft_consensus.cc:2454] T b89e275e1e274042a2cc120111080433 P b550bed5a64f466c8ba5a1761f3bef39 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 543169fd701141c181c55566d2cae00d in term 2.
I20240502 01:56:00.460794 28713 leader_election.cc:302] T b89e275e1e274042a2cc120111080433 P 543169fd701141c181c55566d2cae00d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 543169fd701141c181c55566d2cae00d, b550bed5a64f466c8ba5a1761f3bef39, d50f07af87d74f039fb2b43cf166fa43; no voters: 
I20240502 01:56:00.460947 27849 raft_consensus.cc:2454] T b89e275e1e274042a2cc120111080433 P a64152fd9595478ca767a9f248ef3b85 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 543169fd701141c181c55566d2cae00d in term 2.
I20240502 01:56:00.461077 29823 raft_consensus.cc:2781] T b89e275e1e274042a2cc120111080433 P 543169fd701141c181c55566d2cae00d [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:56:00.461504 29823 raft_consensus.cc:686] T b89e275e1e274042a2cc120111080433 P 543169fd701141c181c55566d2cae00d [term 2 LEADER]: Becoming Leader. State: Replica: 543169fd701141c181c55566d2cae00d, State: Running, Role: LEADER
I20240502 01:56:00.461846 29823 consensus_queue.cc:237] T b89e275e1e274042a2cc120111080433 P 543169fd701141c181c55566d2cae00d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } }
I20240502 01:56:00.464525 27566 catalog_manager.cc:5484] T b89e275e1e274042a2cc120111080433 P 543169fd701141c181c55566d2cae00d reported cstate change: term changed from 1 to 2, leader changed from a64152fd9595478ca767a9f248ef3b85 (127.9.176.194) to 543169fd701141c181c55566d2cae00d (127.9.176.196). New cstate: current_term: 2 leader_uuid: "543169fd701141c181c55566d2cae00d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:56:00.882295 27849 raft_consensus.cc:1261] T b89e275e1e274042a2cc120111080433 P a64152fd9595478ca767a9f248ef3b85 [term 2 FOLLOWER]: Refusing update from remote peer 543169fd701141c181c55566d2cae00d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 01:56:00.883034 29823 consensus_queue.cc:1035] T b89e275e1e274042a2cc120111080433 P 543169fd701141c181c55566d2cae00d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64152fd9595478ca767a9f248ef3b85" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 38689 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:56:00.956480 27715 raft_consensus.cc:1261] T b89e275e1e274042a2cc120111080433 P d50f07af87d74f039fb2b43cf166fa43 [term 2 FOLLOWER]: Refusing update from remote peer 543169fd701141c181c55566d2cae00d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 01:56:00.956965 29823 consensus_queue.cc:1035] T b89e275e1e274042a2cc120111080433 P 543169fd701141c181c55566d2cae00d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:56:00.961292 27981 raft_consensus.cc:1261] T b89e275e1e274042a2cc120111080433 P b550bed5a64f466c8ba5a1761f3bef39 [term 2 FOLLOWER]: Refusing update from remote peer 543169fd701141c181c55566d2cae00d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 01:56:00.962954 30353 consensus_queue.cc:1035] T b89e275e1e274042a2cc120111080433 P 543169fd701141c181c55566d2cae00d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:56:01.353658 28778 consensus_queue.cc:237] T b89e275e1e274042a2cc120111080433 P 543169fd701141c181c55566d2cae00d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } }
I20240502 01:56:01.355499 27981 raft_consensus.cc:1261] T b89e275e1e274042a2cc120111080433 P b550bed5a64f466c8ba5a1761f3bef39 [term 2 FOLLOWER]: Refusing update from remote peer 543169fd701141c181c55566d2cae00d: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 01:56:01.355602 27715 raft_consensus.cc:1261] T b89e275e1e274042a2cc120111080433 P d50f07af87d74f039fb2b43cf166fa43 [term 2 FOLLOWER]: Refusing update from remote peer 543169fd701141c181c55566d2cae00d: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 01:56:01.356186 29823 consensus_queue.cc:1035] T b89e275e1e274042a2cc120111080433 P 543169fd701141c181c55566d2cae00d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:56:01.356472 30353 consensus_queue.cc:1035] T b89e275e1e274042a2cc120111080433 P 543169fd701141c181c55566d2cae00d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:56:01.358899 30336 raft_consensus.cc:2932] T b89e275e1e274042a2cc120111080433 P 543169fd701141c181c55566d2cae00d [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER a64152fd9595478ca767a9f248ef3b85 (127.9.176.194) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } }
I20240502 01:56:01.359450 27715 raft_consensus.cc:2932] T b89e275e1e274042a2cc120111080433 P d50f07af87d74f039fb2b43cf166fa43 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER a64152fd9595478ca767a9f248ef3b85 (127.9.176.194) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } }
I20240502 01:56:01.362828 27558 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b89e275e1e274042a2cc120111080433 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:56:01.364045 27981 raft_consensus.cc:2932] T b89e275e1e274042a2cc120111080433 P b550bed5a64f466c8ba5a1761f3bef39 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER a64152fd9595478ca767a9f248ef3b85 (127.9.176.194) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } } }
I20240502 01:56:01.365706 27570 catalog_manager.cc:5484] T b89e275e1e274042a2cc120111080433 P 543169fd701141c181c55566d2cae00d reported cstate change: config changed from index 6 to 8, VOTER a64152fd9595478ca767a9f248ef3b85 (127.9.176.194) evicted. New cstate: current_term: 2 leader_uuid: "543169fd701141c181c55566d2cae00d" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d50f07af87d74f039fb2b43cf166fa43" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b550bed5a64f466c8ba5a1761f3bef39" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40013 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "543169fd701141c181c55566d2cae00d" member_type: VOTER last_known_addr { host: "127.9.176.196" port: 38773 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:56:01.371088 27827 tablet_service.cc:1514] Processing DeleteTablet for tablet b89e275e1e274042a2cc120111080433 with delete_type TABLET_DATA_TOMBSTONED (TS a64152fd9595478ca767a9f248ef3b85 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:55806
I20240502 01:56:01.371906 30372 tablet_replica.cc:330] T b89e275e1e274042a2cc120111080433 P a64152fd9595478ca767a9f248ef3b85: stopping tablet replica
I20240502 01:56:01.372172 30372 raft_consensus.cc:2229] T b89e275e1e274042a2cc120111080433 P a64152fd9595478ca767a9f248ef3b85 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:56:01.372421 30372 raft_consensus.cc:2258] T b89e275e1e274042a2cc120111080433 P a64152fd9595478ca767a9f248ef3b85 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:56:01.373312 30372 ts_tablet_manager.cc:1875] T b89e275e1e274042a2cc120111080433 P a64152fd9595478ca767a9f248ef3b85: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:56:01.378269 30372 ts_tablet_manager.cc:1888] T b89e275e1e274042a2cc120111080433 P a64152fd9595478ca767a9f248ef3b85: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.7
I20240502 01:56:01.378407 30372 log.cc:1196] T b89e275e1e274042a2cc120111080433 P a64152fd9595478ca767a9f248ef3b85: Deleting WAL directory at /tmp/dist-test-taskJZxTxO/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714614751024524-9923-0/raft_consensus-itest-cluster/ts-1/wal/wals/b89e275e1e274042a2cc120111080433
I20240502 01:56:01.379230 27560 catalog_manager.cc:4830] TS a64152fd9595478ca767a9f248ef3b85 (127.9.176.194:38689): tablet b89e275e1e274042a2cc120111080433 (table rebalance_test_table_2 [id=d490226c88e64345ad42c7ceb86bf959]) successfully deleted
I20240502 01:56:01.708578 28758 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:56:01.739365 27962 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:01.740469 27827 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:01.771073 27696 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:56:01.777530 28626 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:56:01.929322 28758 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:56:01.936672 27827 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:01.939734 27962 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:01.941386 27696 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:56:01.944782 28626 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:56:02.112109 28758 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:56:02.132602 27827 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:02.135056 27696 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:56:02.140960 28626 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:56:02.141078 27962 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:02.323841 28758 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:56:02.324733 27827 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:02.335350 28626 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:56:02.347649 27696 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:56:02.373471 27962 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:02.519493 28758 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:56:02.529219 27827 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:02.530500 27962 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:02.553269 27696 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:56:02.555080 28626 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:56:02.766105  9923 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu pid 27634
I20240502 01:56:02.775532 27757 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:56:02.946067  9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu with pid 27634
W20240502 01:56:02.974539 28580 connection.cc:535] server connection from 127.9.176.193:40429 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 01:56:02.975554  9923 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu pid 27766
I20240502 01:56:02.981781 27888 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 01:56:03.002576 28579 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.176.193:34203: connect: Connection refused (error 111)
W20240502 01:56:03.004215 28579 consensus_peers.cc:485] T d404c85b11594e7eb1257fcf384beafa P c541378ca18944c49c37f49097bc7f81 -> Peer d50f07af87d74f039fb2b43cf166fa43 (127.9.176.193:34203): Couldn't send request to peer d50f07af87d74f039fb2b43cf166fa43. Status: Network error: Client connection negotiation failed: client connection to 127.9.176.193:34203: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:56:03.047875 28712 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.176.193:34203: connect: Connection refused (error 111)
W20240502 01:56:03.049324 28712 consensus_peers.cc:485] T b89e275e1e274042a2cc120111080433 P 543169fd701141c181c55566d2cae00d -> Peer d50f07af87d74f039fb2b43cf166fa43 (127.9.176.193:34203): Couldn't send request to peer d50f07af87d74f039fb2b43cf166fa43. Status: Network error: Client connection negotiation failed: client connection to 127.9.176.193:34203: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:56:03.130499 28579 consensus_peers.cc:485] T 63136d6964084523bb950cc6568a273b P c541378ca18944c49c37f49097bc7f81 -> Peer d50f07af87d74f039fb2b43cf166fa43 (127.9.176.193:34203): Couldn't send request to peer d50f07af87d74f039fb2b43cf166fa43. Status: Network error: Client connection negotiation failed: client connection to 127.9.176.193:34203: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:56:03.157594  9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu with pid 27766
I20240502 01:56:03.198971  9923 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu pid 27899
I20240502 01:56:03.205080 28023 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:56:03.354362  9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu with pid 27899
W20240502 01:56:03.382920 28580 connection.cc:535] client connection to 127.9.176.195:40013 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 01:56:03.383041  9923 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu pid 28696
W20240502 01:56:03.384421 28580 consensus_peers.cc:485] T 5e8b27dca9af473188c50269820343f1 P c541378ca18944c49c37f49097bc7f81 -> Peer b550bed5a64f466c8ba5a1761f3bef39 (127.9.176.195:40013): Couldn't send request to peer b550bed5a64f466c8ba5a1761f3bef39. Status: Network error: Client connection negotiation failed: client connection to 127.9.176.195:40013: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:56:03.385535 28713 consensus_peers.cc:485] T 31de6c641c094cce894d8513b878350c P 543169fd701141c181c55566d2cae00d -> Peer b550bed5a64f466c8ba5a1761f3bef39 (127.9.176.195:40013): Couldn't send request to peer b550bed5a64f466c8ba5a1761f3bef39. Status: Network error: Client connection negotiation failed: client connection to 127.9.176.195:40013: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:56:03.390039 28819 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:56:03.532747  9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu with pid 28696
W20240502 01:56:03.540402 28580 consensus_peers.cc:485] T 63136d6964084523bb950cc6568a273b P c541378ca18944c49c37f49097bc7f81 -> Peer b550bed5a64f466c8ba5a1761f3bef39 (127.9.176.195:40013): Couldn't send request to peer b550bed5a64f466c8ba5a1761f3bef39. Status: Network error: Client connection negotiation failed: client connection to 127.9.176.195:40013: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:56:03.559494  9923 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu pid 28563
I20240502 01:56:03.565699 28687 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:56:03.708508  9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu with pid 28563
I20240502 01:56:03.730582  9923 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu pid 27543
I20240502 01:56:03.731791 27604 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:56:03.897831  9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJZxTxO/build/asan/bin/kudu with pid 27543
2024-05-02T01:56:03Z chronyd exiting
[       OK ] LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer/idx2_rf3_t3_l_3_2 (30785 ms)
[----------] 2 tests from LocationAwareRebalancingParamTest (53680 ms total)

[----------] Global test environment tear-down
[==========] 7 tests from 6 test suites ran. (212911 ms total)
[  PASSED  ] 6 tests.
[  FAILED  ] 1 test, listed below:
[  FAILED  ] ConcurrentRebalancersTest.TwoConcurrentRebalancers/1, where GetParam() = 4-byte object <01-00 00-00>

 1 FAILED TEST